WebSocket protocol

The ingestion WebSocket is how audio reaches a session. You get its coordinates from POST /api/sessions or POST /api/sessions/{id}/reconnect - the ingestion object carries url, wsToken, and ready.

Connecting

URL: ingestion.url, in the form wss://<worker-host>/ws/{session_id}.
Auth: offer the token as a WebSocket subprotocol - Sec-WebSocket-Protocol: facemode.<wsToken>. The server echoes that subprotocol in the handshake.
Query fallback: clients that cannot set subprotocols may pass ?token=<wsToken> on the URL instead.
Compression: do not negotiate permessage-deflate; it is disabled.

Token rules:

wsToken is a signed credential with a 15 minute expiry and is single-use - the first successful connection consumes it.
Only one WebSocket may be active per session; a second concurrent connect is rejected.
After it expires or is consumed, call reconnect for a fresh one.

Pre-upgrade failures

If the handshake is rejected, the socket is closed before any application traffic. The rejection carries an application close code - your WebSocket client surfaces it as a failed upgrade or an immediate close, depending on the library:

Close code	Cause
`4000`	Required claim missing from the token
`4001`	Missing, malformed, expired, or already-used token; session ID in the token does not match the URL
`4403`	`Origin` header not allowed (browser clients, when origin rules are configured)
`4004`	Unknown session ID
`4409`	Session already has an active connection, or it is not ready for a reconnect yet
`4410`	Session is ending
`4429`	Too many connections from your IP
`4503`	Session is not ready to admit connections yet - retry shortly
`1011`	Server-side configuration error - contact support

Handshake

The first application message must be start; the server answers started. Until started arrives, do not send audio, utterance control, or pings - anything else first fails with NOT_NEGOTIATED.

// client ->
{"type":"start","session_id":"6ba7b810-9dad-11d1-80b4-00c04fd430c8","audio_encoding":"pcm_s16le","sample_rate":16000,"channels":1}

// <- server
{"type":"started","session_id":"6ba7b810-9dad-11d1-80b4-00c04fd430c8","server_sample_rate":48000,"server_channels":1}

server_sample_rate/server_channels describe FaceMode’s internal format - informational only; keep sending audio in the format you negotiated.

Client -> server messages

All control messages are JSON text frames. PCM audio may also travel as raw binary frames.

Type	Fields	Notes
`start`	`session_id` (string, required), `audio_encoding` (string, required), `sample_rate` (int, required), `channels` (int, required), `avatar_id` (string, optional), `metadata` (object, optional), `utterance_silence_ms` (int, optional)	First message only; sends it a second time fails with `INVALID_FORMAT`
binary frame	raw PCM bytes	Only for `pcm_s16le`/`pcm_f32le` encodings; payload must be channel-aligned
`audio`	`data` (base64 string, required), `seq` (int, required)	Only for `base64_json` sessions
`start_utterance`	`seq` (int, required)	Explicitly open an utterance
`end_utterance`	`seq` (int, required)	Close the utterance; the server drains queued speech then replies `utterance_ended`
`cancel_utterance`	`seq` (int, required)	Abort the utterance; queued audio is discarded
`end_session`	`seq` (int, required)	Graceful end; the server replies `ended`
`ping`	`ts` (number, required, client timestamp in ms)	Keepalive; only valid after `started`

`start` field details

audio_encoding: pcm_s16le, pcm_f32le, or base64_json.
sample_rate: 8000-48000 Hz. Preferred values: 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000. Other in-range rates work but are resampled.
channels: 1 or 2.
metadata: free-form object. For base64_json, metadata.inner_encoding: "pcm_f32le" opts into float PCM inside the base64 payload (default is 16-bit PCM).
utterance_silence_ms: 0 disables silence-based utterance detection; 100-5000 tunes the gap. See below.

Server -> client messages

Type	Fields	Meaning
`started`	`session_id`, `server_sample_rate`, `server_channels`	Handshake ack; media may now flow
`audio_ready`	`seq`	The session’s outbound audio path is live (sent once, `seq` `0`; repeated after `started` on reconnect if already live)
`utterance_ended`	`seq`, `cycles`	Utterance fully rendered and drained. `seq` echoes your `end_utterance` seq; `cycles` counts drain iterations (diagnostic)
`ended`	`session_id`	Protocol ack for `end_session`. See below - this is not a cleanup guarantee
`pong`	`ts`, `server_ts`	Echoes your `ts`, plus `server_ts` (server ms timestamp) for RTT measurement
`error`	`code`, `message`, `fatal`	`fatal: true` means the server will close the socket after sending

Sequence numbers

seq is a non-negative integer maintained by the client.
Sequences must not move backwards: every sequenced message (audio, start_utterance, end_utterance, cancel_utterance, end_session) must have seq >= the last one sent. Violations fail with INVALID_FORMAT.
Binary PCM frames carry no seq.
utterance_ended.seq echoes the end_utterance seq it answers - use it to match requests to drains. In inferred mode (below), the seq is assigned by the server.

Utterance boundaries: inferred vs explicit

FaceMode needs to know where one utterance ends so it can flush speech to avatar video. Two modes:

Inferred (default): the first audio frame opens an utterance; a silence gap of utterance_silence_ms (server default ~800 ms) closes it and produces utterance_ended. Best for simple pipelines - just stream audio.
Explicit: drive start_utterance/end_utterance/cancel_utterance yourself for tight control (for example, aligning with your own TTS events).

Sending any explicit utterance message latches the session into explicit mode permanently - do not mix the two styles on one connection. Sessions created with inputProvider deepgram or gemini start in provider-driven (explicit) mode.

Set utterance_silence_ms in start to tune inferred mode: 0 turns inference off (you must then use explicit control), 100-5000 sets the silence gap in milliseconds.

Heartbeats and timeouts

After started, send ping periodically (for example every 15 s) and expect pong. One ping in flight at a time is enough.
The server closes idle sockets: prolonged silence with no frames ends the connection (close code 1000, reason Inactivity timeout).
The REST-level POST /api/sessions/{id}/heartbeat is separate bookkeeping - use both.

Errors and close semantics

error frames carry {code, message, fatal}:

Code	Meaning
`INVALID_FORMAT`	Malformed JSON, bad field types/values, channel-misaligned PCM, unsupported message type
`NOT_NEGOTIATED`	Media or control sent before `start`/`started`
`SESSION_NOT_FOUND`	`start.session_id` does not match this connection’s session (fatal)
`RATE_LIMITED`	Audio throughput exceeded the per-connection budget (fatal)
`INTERNAL_ERROR`	Server-side failure (fatal)
`AVATAR_NOT_READY`	Avatar assets were not ready to render

Fatal errors are followed by a server-initiated close. Close codes used after upgrade (for handshake-time rejections see Pre-upgrade failures above - note 4000/4001/4004 are reused with different meanings):

Code	Meaning
`1000`	Inactivity timeout
`4000`	Generic fatal protocol error / oversized payload / internal handler error
`4001`	Credential or ownership-claim failure during reconnect
`4004`	`SESSION_NOT_FOUND`
`4005`	`NOT_NEGOTIATED`
`4006`	`RATE_LIMITED`

ended acknowledges end_session at the protocol level only - it does not prove session cleanup finished. To confirm the session is fully torn down, poll GET /api/sessions/{id} until session.status is ENDED, or call POST /api/sessions/{id}/end.

Reconnecting

When the socket dies mid-session:

Call POST /api/sessions/{id}/reconnect for a fresh ingestion.url + wsToken (old tokens cannot be reused).
Open the new WebSocket with the facemode.<wsToken> subprotocol.
Re-send start - the new connection performs a fresh handshake. If audio was already flowing, audio_ready follows started again.
Resume your seq counter or restart at 0 - sequence state is per connection.

API Reference

Sessions

Avatars

API Keys

WebSocket protocol

Connecting

Pre-upgrade failures

Handshake

Client -> server messages

`start` field details

Server -> client messages

Sequence numbers

Utterance boundaries: inferred vs explicit

Heartbeats and timeouts

Errors and close semantics

Reconnecting

​Connecting

​Pre-upgrade failures

​Handshake

​Client -> server messages

​start field details

​Server -> client messages

​Sequence numbers

​Utterance boundaries: inferred vs explicit

​Heartbeats and timeouts

​Errors and close semantics

​Reconnecting

Connecting

Pre-upgrade failures

Handshake

Client -> server messages

`start` field details

Server -> client messages

Sequence numbers

Utterance boundaries: inferred vs explicit

Heartbeats and timeouts

Errors and close semantics

Reconnecting