Skip to main content
FaceMode is a real-time AI talking-head service: your backend streams audio in, and a lip-synced avatar video track comes out the other side, published into a WebRTC room you already own. Your frontend renders that track with the standard LiveKit or Daily viewer SDK, so FaceMode slots into your existing stack as a drop-in avatar layer.

​
The big picture

There are three separate connections, each owned by a different part of your system:
FaceMode never sits between your frontend and your room. FaceMode joins your room as a publishing participant; your viewers subscribe to its tracks exactly as they would any other participant.

​
What happens inside a session

When you create a session, FaceMode assigns a dedicated avatar session to it. From your audio to the viewer’s screen, the session pipeline looks like this:
1

Audio normalization

Every chunk you send is decoded using the format you negotiated in the start message (pcm_s16le, pcm_f32le, or base64_json) and resampled to a canonical internal format. You can send any integer sample rate from 8000 to 48000 Hz; you do not need to resample client-side.
2

Frame generation

A talking-head model generates lip-synced avatar frames at 512x512. Every published frame is upscaled to 1024x1024 before encoding, so viewers receive a sharp 1024x1024 stream.
3

Encode and publish

Frames are encoded as VP8 or H264 video and published into your room along with a matching audio track. The result is a single avatar participant your frontend subscribes to.

​
Idle and live states

The avatar is always on screen, even when nobody is speaking:
  • Idle - between utterances, FaceMode streams a natural idle loop with subtle motion instead of a frozen face.
  • Live - while audio is flowing, generated lip-synced frames replace the idle loop.
  • Crossfade - transitions between idle and live are blended, so the avatar never pops or cuts when speech starts or stops.
This means your frontend does not need any state handling: the same avatar video track carries idle motion, speech, and the transitions between them.

​
Latency

FaceMode targets 960 ms from audio arriving at FaceMode to the corresponding frame hitting the room. End-to-end latency your users perceive adds your upstream audio latency (TTS generation, network to FaceMode) and the viewer’s WebRTC path on top. To stay close to the target:
  • Stream audio in small, regular chunks (about 100 ms of PCM per message is a good default).
  • Keep the ingestion WebSocket open for the life of the session instead of reconnecting per utterance.
  • Open the socket as soon as ingestion.ready is true so cold-start time overlaps your own setup work.

​
Who owns which credential

Four credentials appear in a session. Three of them must never leave your server.
The room token grants publish permission, and the wsToken controls the ingestion socket. If either reaches a browser, anyone can publish into your room or hijack the session’s audio. Browsers only ever need a subscribe-only viewer token.

​
Where to go next