Session lifecycle

A FaceMode session is one avatar streaming into one room. It has two halves that you manage together: a control half on the REST API (create, poll, reconnect, end) and a media half on a per-session ingestion WebSocket (negotiate once, stream audio, receive events).

Two kinds of state

Every session response exposes two fields:

status - the session’s own state. It is ACTIVE from creation until the session is fully cleaned up, when it becomes ENDED.
workerStatus - the state of the avatar session provisioned for it.

`workerStatus`	Meaning
`PENDING`	Session created; provisioning is being prepared.
`ASSIGNING`	A dedicated avatar session is being allocated.
`ASSIGNED`	The avatar session is assigned and starting up.
`ACTIVE`	The session is live: connected, negotiated, and streaming.
`ENDING`	Termination was requested; cleanup is in progress.
`ENDED`	Terminal. Session cleanup finished and the session was released.
`FAILED`	Terminal. The session failed and cannot be resumed.

ENDED and FAILED are terminal. Once workerStatus reaches either, the ingestion socket is gone for good - create a new session instead of retrying.

Sequence of a healthy session

Step by step

Create the session

POST /api/sessions with avatarId, a room object, and an optional inputProvider. A healthy create returns 201 and usually includes ingestion.ready: true with url and wsToken immediately - session creation is connect-first.

{
  "session": { "id": "7f3c...", "status": "ACTIVE" },
  "workerStatus": "PENDING",
  "ingestion": {
    "ready": true,
    "authority": "websocket",
    "url": "wss://<worker-host>/ws/7f3c...",
    "wsToken": "eyJhbGciOi..."
  }
}

Wait for ingestion readiness

If ingestion.ready is false, poll GET /api/sessions/:id with bounded backoff until ingestion.ready is true. A practical budget is about 240 seconds total for cold starts, starting near 250 ms and capping the retry interval around 2 seconds. Abort early if workerStatus becomes FAILED or ENDED.

Open the ingestion WebSocket

Connect to ingestion.url with the subprotocol facemode.<wsToken>. The token is a one-time, 15-minute JWT: a consumed token is never accepted again, so each socket needs its own token. Clients that cannot set subprotocols may pass the same token as a ?token= query parameter. If the response includes ingestion.headers, forward them on the upgrade request.Budget up to about 240 seconds for the handshake so a cold-starting session has time to accept the connection, and disable permessage-deflate on the socket. The official plugins use a 240-second open budget.

const ws = new WebSocket(ingestion.url, [`facemode.${ingestion.wsToken}`], {
  perMessageDeflate: false,
  handshakeTimeout: 240_000,
  headers: ingestion.headers ?? {},
});

Negotiate with start

The first application message on the socket must be start. It pins the audio format for the life of the session.

{
  "type": "start",
  "session_id": "7f3c...",
  "audio_encoding": "pcm_s16le",
  "sample_rate": 16000,
  "channels": 1,
  "utterance_silence_ms": 800
}

The session answers started:

{ "type": "started", "session_id": "7f3c...", "server_sample_rate": 48000, "server_channels": 1 }

Stream utterances

Send audio as binary PCM frames, or as audio JSON messages when you negotiated base64_json. See Utterance flow below for inferred vs explicit mode.

Keep the connection healthy

After started, send an application ping roughly every 15 seconds and expect pong back. This is a transport heartbeat only - it does not count as session activity. A POST /api/sessions/:id/heartbeat call is also available to refresh the session’s activity timestamp from the control plane.

End the session

Send end_session, wait for ended, then close the socket and call POST /api/sessions/:id/end. See Ending and cleanup for why the ended message alone is not enough.

Utterance flow

The protocol supports two utterance modes. Pick one per session - sending any explicit utterance control permanently latches the session into explicit mode.

Inferred mode (recommended for simple pipelines)

Send start, then stream audio. FaceMode watches for a silence gap and closes the utterance itself, emitting audio_ready when speech begins to render and utterance_ended after the gap. Tune the gap with utterance_silence_ms in start: 0 disables inference, 100 to 5000 sets the silence window, and the default is 800 ms. Never send start_utterance, end_utterance, or cancel_utterance in this mode.

Explicit mode

Bracket each utterance yourself: start_utterance { seq }, audio frames, then end_utterance { seq }. FaceMode replies utterance_ended { seq, cycles } when the utterance has drained - seq matches your end_utterance and cycles counts the inference cycles consumed. Use cancel_utterance { seq } to abort an in-flight utterance, for example on barge-in. Keep seq monotonically increasing across the session, including across reconnects.

Message reference

Client to server:

Message	Shape	Notes
`start`	`{ session_id, audio_encoding, sample_rate, channels, avatar_id?, metadata?, utterance_silence_ms? }`	First message; pins the format.
binary audio	raw PCM frame	For `pcm_s16le` / `pcm_f32le`.
`audio`	`{ data, seq }`	Base64 payload for `base64_json`.
`start_utterance`	`{ seq }`	Explicit mode only.
`end_utterance`	`{ seq }`	Explicit mode; awaits `utterance_ended`.
`cancel_utterance`	`{ seq }`	Explicit mode; drops in-flight audio.
`end_session`	`{ seq }`	Awaits `ended`.
`ping`	`{ ts }`	Only after `started`.

Server to client:

Message	Shape	Notes
`started`	`{ session_id, server_sample_rate, server_channels }`	Negotiation complete.
`audio_ready`	`{ seq }`	The session is producing audio for the utterance.
`utterance_ended`	`{ seq, cycles }`	Utterance drained.
`ended`	`{ session_id }`	`end_session` acknowledged.
`pong`	`{ ts, server_ts }`	Heartbeat reply; compare `ts` to measure RTT.
`session_ending`	`{ reason }`	Server-initiated shutdown; the socket closes next.
`error`	`{ code, message, fatal }`	See error codes below.

Error codes

Code	Meaning	Recoverable?
`INVALID_FORMAT`	Malformed message or bad field value	No - fix the payload
`NOT_NEGOTIATED`	Audio sent before `started`	Send `start` first
`SESSION_NOT_FOUND`	Unknown or expired session	No - create a new session
`RATE_LIMITED`	Sending too fast	Slow down and retry
`INTERNAL_ERROR`	Server-side failure	Usually fatal
`AVATAR_NOT_READY`	Avatar pipeline still loading	Wait for `started`/retry

Errors carry fatal: true when the socket is about to close; treat fatal errors as end-of-session.

Reconnecting a dropped transport

If the WebSocket drops mid-session, do not create a new session - call POST /api/sessions/:id/reconnect instead:

POST /api/sessions/:id/reconnect
x-api-key: fm_...
Content-Type: application/json

{ "room": { "type": "livekit", "url": "wss://...", "token": "..." } }

The response returns a fresh ingestion block with a new one-time wsToken. Reuse the same reconnect budget you used for the first connect.
room is optional and only needed if you are rotating room credentials. Its type must match the session’s original room type, and inputProvider (if sent) must match the value used at creation.
Open a new socket, resend the identical start message, and wait for started before resuming audio. Keep seq numbers monotonic across the reconnect.
Bound your attempts (two or three retries with short backoff is typical) and end the session if reconnects are exhausted.

A dropped socket does not end the session. FaceMode keeps the session warm briefly so a reconnect can reattach cleanly - but audio sent during the gap is lost, so pause upstream until started arrives on the new socket.

Ending and cleanup

Ending has two layers, and conflating them causes leaked sessions:

Protocol layer - end_session on the socket is answered by ended. This is only an acknowledgement that FaceMode received your request. It is not proof that backend cleanup finished.
Control layer - POST /api/sessions/:id/end finalizes the session. It returns 200 with pending: false when cleanup completed inline, or 202 with pending: true while session cleanup is still in flight. A 400 with SESSION_ALREADY_ENDED means a previous request already finalized it - treat that as success, not failure.

After a 202, poll GET /api/sessions/:id every half-second or so until status is ENDED (or workerStatus reaches ENDED/FAILED). A practical budget is about 45 seconds. FaceMode can also end the session on its own: it sends session_ending with a reason, then closes the socket with code 1000. Sessions with no audio activity for an extended idle period are ended automatically, so explicit cleanup on your side is always the safer path. The reason is available as workerAssignment.terminationReason on status reads. A session in ENDING no longer blocks a new POST /api/sessions - the concurrency slot is freed when termination is requested.

Always end sessions you no longer need. An abandoned session keeps its avatar - and its room participant - alive until the idle policy reclaims it.

How FaceMode works - the architecture this lifecycle drives.
Input providers - choosing audio_encoding, sample_rate, and channels for start.
Rooms and egress - the room object used at create and reconnect.

Get Started

Concepts

Integrations

Session lifecycle

Two kinds of state

Sequence of a healthy session

Step by step

Utterance flow

Message reference

Error codes

Reconnecting a dropped transport

Ending and cleanup

​Two kinds of state

​Sequence of a healthy session

​Step by step

​Utterance flow

​Message reference

​Error codes

​Reconnecting a dropped transport

​Ending and cleanup

​Related pages

Two kinds of state

Sequence of a healthy session

Step by step

Utterance flow

Message reference

Error codes

Reconnecting a dropped transport

Ending and cleanup

Related pages