Pipecat

The pipecat-facemode package is a small Pipecat FrameProcessor that taps TTSAudioRawFrame, sends the PCM audio to FaceMode over the canonical WebSocket protocol, and converts the avatar video and audio tracks in your LiveKit room back into Pipecat output frames. It does not replace a Pipecat transport.

Pipeline placement

Place FaceModeVideoService after your TTS service and before the output transport:

transport input -> STT -> LLM -> TTS -> FaceModeVideoService -> transport output

Do not put another audio output before this service. By default the processor consumes TTSAudioRawFrame so the source TTS audio and the avatar’s returned audio are not both played.

Install

pip install pipecat-facemode

The package supports Pipecat 1.7 through the current 1.x line, LiveKit RTC 1.x, aiohttp 3.x, websockets 14 through 16, and NumPy 2.x. The websockets>=14 requirement matters: the service uses the additional_headers handshake API and never falls back to an unaffinitized connection.

Quickstart

This example uses Pipecat’s LiveKit transport. Because FaceMode publishes the avatar video and audio tracks directly into the room, the transport’s own audio and video outputs are disabled.

import os

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.transports.livekit.transport import LiveKitParams, LiveKitTransport
from pipecat.workers.runner import WorkerRunner
from pipecat.frames.frames import EndFrame, TTSSpeakFrame

from pipecat_facemode import FaceModeVideoService


async def main() -> None:
    transport = LiveKitTransport(
        url=os.environ["LIVEKIT_URL"],
        token=os.environ["LIVEKIT_TRANSPORT_TOKEN"],
        room_name=os.environ["LIVEKIT_ROOM_NAME"],
        params=LiveKitParams(
            audio_in_enabled=True,
            audio_out_enabled=False,
            video_out_enabled=False,
            video_out_is_live=True,
        ),
    )

    tts = CartesiaTTSService(
        api_key=os.environ["CARTESIA_API_KEY"],
        settings=CartesiaTTSService.Settings(
            voice="71a7ad14-091c-4e8e-a314-022ece01c121",
        ),
    )

    facemode = FaceModeVideoService(
        api_key=os.environ["FACEMODE_API_KEY"],
        avatar_id=os.environ.get("FACEMODE_AVATAR_ID", ""),
        api_url=os.environ.get("FACEMODE_API_URL", "https://api.facemode.io/api"),
        room={
            "type": "livekit",
            "url": os.environ["LIVEKIT_URL"],
            "token": os.environ["LIVEKIT_PUBLISH_TOKEN"],
        },
        livekit_subscriber_token=os.environ["LIVEKIT_SUBSCRIBER_TOKEN"],
        avatar_participant_identity=os.environ.get(
            "FACEMODE_AVATAR_PARTICIPANT_IDENTITY", "facemode-avatar"
        ),
        input_provider=os.environ.get("FACEMODE_INPUT_PROVIDER") or None,
        room_name=os.environ["LIVEKIT_ROOM_NAME"],
    )

    pipeline = Pipeline([
        transport.input(),
        tts,
        facemode,
        transport.output(),
    ])
    worker = PipelineWorker(
        pipeline,
        params=PipelineParams(audio_out_sample_rate=24_000),
    )

    @transport.event_handler("on_first_participant_joined")
    async def on_first_participant_joined(_transport, _participant_id) -> None:
        await worker.queue_frames(
            [TTSSpeakFrame("Hello from a Pipecat pipeline connected to FaceMode.")]
        )

    @transport.event_handler("on_participant_disconnected")
    async def on_participant_disconnected(_transport, _participant_id) -> None:
        await worker.queue_frame(EndFrame())

    await WorkerRunner().run(worker)

Constructor options

Option	Default	Description
`api_key`	required	FaceMode API key. Required.
`room`	required	Room object (`LiveKitRoom` or mapping) with `type`, `url`, and a server-side `token` that lets FaceMode join and publish.
`avatar_id`	`""`	FaceMode avatar ID.
`api_url`	`https://api.facemode.io/api`	FaceMode API root. Override for self-hosted or local backends.
`room_name`	`None`	LiveKit room name. If omitted, it is read from the connected room or `room.name`.
`livekit_subscriber_token`	falls back to `room.token`	Token this process uses to subscribe to the room. Supply a separate token in production.
`avatar_participant_identity`	`"facemode-avatar"`	Identity filter for avatar tracks. Set it to the identity in the publish token when that token uses a different identity.
`input_provider`	`None`	Optional session input provider: `deepgram`, `gemini`, `gnani`, `elevenlabs`, `openai`, `cartesia`, `sarvam`, or `custom`. Persisted for the session and reused on reconnect.
`forward_tts_audio`	`False`	Also pass source `TTSAudioRawFrame` frames downstream. Enable only when the pipeline intentionally needs both audio streams.
`utterance_idle_timeout`	`0.35`	Seconds of TTS silence after which the service sends `end_utterance`.
`protocol_timeout`	`15.0`	Seconds to wait for the `started` acknowledgement.
`api_timeout`	`30.0`	REST request timeout in seconds.
`http_session`	`None`	Optional caller-owned `aiohttp.ClientSession`.

FaceModeService and FaceModeProcessor are exported aliases for FaceModeVideoService. Useful members: service.session_id, service.is_started, and service.avatar_ready.

How audio flows

Each TTSAudioRawFrame is sent to FaceMode as binary little-endian signed 16-bit PCM, with no JSON or base64 wrapper. start_utterance precedes the first chunk of an utterance.
end_utterance is sent on a TTS stop frame or after utterance_idle_timeout seconds of silence. cancel_utterance is sent on Pipecat interruption frames.
The avatar’s LiveKit audio track is pushed downstream as OutputAudioRawFrame; the avatar video track becomes an output image frame.
The negotiated sample rate must be an integer between 8000 and 48000 Hz (preferred: 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000) and channels must be 1 or 2. A mid-session format change raises FaceModeProtocolError.

Token grants

Keep all LiveKit and FaceMode credentials on the server. Two LiveKit connections are involved, and production deployments should use separate participant identities and tokens for each:

room.token identifies the avatar participant and must grant room_join for the exact room name, can_publish (for the avatar video and audio tracks), and can_subscribe.
livekit_subscriber_token identifies this Pipecat process and must grant room_join for the same room and can_subscribe. Grant publish only if the application needs it.

A token with publish and subscribe grants can be reused for both connections, but do not share a participant identity between them: a LiveKit room permits only one participant per identity. Example server-side token generation with livekit-api:

from livekit import api

publish_token = (
    api.AccessToken()
    .with_identity("facemode-avatar")
    .with_name("FaceMode Avatar")
    .with_grants(
        api.VideoGrants(
            room_join=True,
            room=room_name,
            can_publish=True,
            can_subscribe=True,
        )
    )
    .to_jwt()
)

subscriber_token = (
    api.AccessToken()
    .with_identity("pipecat-facemode-subscriber")
    .with_name("Pipecat FaceMode Subscriber")
    .with_grants(
        api.VideoGrants(
            room_join=True,
            room=room_name,
            can_subscribe=True,
        )
    )
    .to_jwt()
)

Token generation belongs on your backend only. The LiveKit API secret must never be sent to a client.

Waiting for avatar media

await service.wait_for_avatar() resolves when FaceMode reports audio_ready or the first avatar video frame arrives. It raises FaceModeNotReadyError after the timeout (30 seconds by default; pass timeout=None to wait indefinitely). Call it before announcing that avatar media is available to the user.

Automatic reconnect

If the WebSocket drops after the session is fully started, the service reconnects without a pipeline-visible failure:

It calls POST /api/sessions/{id}/reconnect with the original room object (plus the persisted inputProvider when configured). Every response mints a fresh one-time wsToken; tokens are never reused.
It opens a new socket with the facemode.<newToken> subprotocol, resends the same start negotiation, and waits for started before audio resumes. An explicitly open utterance is re-declared on the new socket.
At most 2 attempts run with bounded backoff; recovery is single-flight. Sequence numbers keep increasing and already-sent audio is never replayed. Sends that arrive during the gap wait for the new socket instead of failing.

No reconnect is attempted on intentional shutdown (EndFrame, CancelFrame, stop, cleanup), on server-terminal states (session_ending, ended, fatal error messages), or before the session is fully started. When the budget is exhausted the failure surfaces as a FaceModeProtocolError.

Lifecycle and failure notes

EndFrame triggers a graceful shutdown: the service sends end_session and waits up to 3 seconds for ended before closing the socket. CancelFrame and cleanup() skip the graceful message.
ended is a WebSocket protocol acknowledgement, not proof that backend session cleanup is complete. Callers that need backend cleanup confirmation should poll GET /api/sessions/{id} until the session reaches a terminal state.
The ingestion-ready poll budget and the WebSocket open budget are each 240 seconds.
LiveKit tracks published before the subscriber connects are discovered at startup; later track_subscribed events are handled automatically. The local subscriber’s own track is always ignored, and remote tracks are accepted only from avatar_participant_identity.
Typed FaceMode exceptions never include API keys or token values.

Error types

All errors derive from FaceModeError (a RuntimeError):

Exception	Raised when
`FaceModeConfigurationError`	Constructor arguments are missing or invalid (bad `api_key`, `room`, `input_provider`, timeouts).
`FaceModeAPIError`	A FaceMode REST call fails or the session enters a `FAILED` / `ENDED` state.
`FaceModeProtocolError`	The WebSocket protocol fails or reconnect attempts are exhausted.
`FaceModeNotReadyError`	The LiveKit connection fails or `wait_for_avatar()` times out.

from pipecat_facemode import (
    FaceModeAPIError,
    FaceModeConfigurationError,
    FaceModeNotReadyError,
    FaceModeProtocolError,
)

Get Started

Concepts

Integrations

Pipeline placement

Install

Quickstart

Constructor options

How audio flows

Token grants

Waiting for avatar media

Automatic reconnect

Lifecycle and failure notes

Error types

​Pipeline placement

​Install

​Quickstart

​Constructor options

​How audio flows

​Token grants

​Waiting for avatar media

​Automatic reconnect

​Lifecycle and failure notes

​Error types

Pipeline placement

Install

Quickstart

Constructor options

How audio flows

Token grants

Waiting for avatar media

Automatic reconnect

Lifecycle and failure notes

Error types