POST /api/sessions accepts an optional inputProvider field that tells the
session which upstream provider is producing your audio. Set it once at
creation: the value is persisted on the session, and a reconnect that sends a
different value is rejected, so treat it as immutable for the session’s life.
inputProvider is optional. Omit it (or use custom) when you stream raw
PCM from your own pipeline - the session still speaks the same canonical
protocol either way.Supported values
The wire formats above are the common defaults for each provider - they are
what the reference integrations use - not a per-provider restriction. The
session accepts the full negotiated format matrix below regardless of which
inputProvider you set.
deepgram and gemini enable a provider profile on the session that
understands that provider’s event stream (turn boundaries, barge-in). The
other values are descriptive hints for the session and use the standard
start plus PCM flow.Audio format matrix
You pin the format once in thestart message; it cannot change mid-session.
FaceMode normalizes everything you send to its canonical internal format, so
you do not need client-side resampling. If your provider gives you a
container (WAV, MP3), strip the headers and send raw PCM samples.
Chunking
Send audio in small, steady chunks rather than large bursts. About 100 ms of PCM per message is the typical cadence:
Formula:
sample_rate * channels * bytes_per_sample * 0.1. Double it for
stereo or pcm_f32le (4 bytes per sample). Chunkier sends (hundreds of
milliseconds at once) add latency; tiny sends add overhead. If your provider
delivers irregular chunk sizes, buffer and re-slice to ~100 ms frames.
Choosing an encoding
pcm_s16le- the default choice. Signed 16-bit little-endian PCM sent as binary WebSocket frames; every major TTS provider can emit it.pcm_f32le- 32-bit float PCM when your pipeline already works in floats (common with Web Audio or DSP code) and you want to skip the conversion.base64_json- base64 audio inside JSONaudio { data, seq }messages. Use it when your source already produces base64 audio events (Gemini-style payloads) or when binary frames are awkward in your stack. It costs ~33% more bytes on the wire, so prefer binary when you have the choice.
Related pages
- Session lifecycle - where
startfits the protocol. - Rooms and egress - the other half of the create request.