API · Render mode

Streaming protocol

One WebSocket per session, from your server: you send the avatar's speech and commands as JSON, and it tells you when each utterance starts and ends on screen.

Connect

WSSws_url from POST /v1/sessions, as given (…/ws/render/1?token=…)

•Add &sample_rate=24000 to choose the audio rate (see below), or send session.configure before the first audio.
•The token can also go in an Authorization: Bearer header instead of the address.
•Wait for session.state_updated with "state": "connected", then send.
•One socket per session: a new connection replaces the old one (the old one closes with 4002).

Audio format

•PCM, signed 16-bit little-endian, mono.
•16000, 22050, 24000 (default), 44100 or 48000 Hz.
•Pieces of any size, about 20 ms to 1 s, as base64 in agent.speak or as raw binary frames. At most 1 MB per frame.
•Send faster than real time. At most 120 seconds of audio may wait to play; more is refused with queue_full.
•Whole samples only: if your speech engine streams odd-length chunks, carry the odd byte over to the next piece.

Commands you send

type	Payload	What it does
`agent.speak`	`{"audio": base64, "text"?: "…"}`	Adds audio to the current utterance. The first piece after a `speak_end` or `interrupt` opens a new one. `text` is what the audio says; it picks the expressions.
(binary frame)	raw PCM	Same as `agent.speak` without text.
`agent.speak_end`	`{"audio"?: base64}`	Seals the utterance. Utterances play one after another.
`agent.speak_text`	`{"text": "…", "cue"?: "warmth"}`	We voice the text with the agent's voice and play it in order with your audio. Up to 5000 characters; expression tags are allowed and never spoken.
`agent.interrupt`	—	Stops what is playing and drops everything queued, voice and lips together.
`agent.start_listening`	—	The listening look between utterances. agent.stop_listening ends it.
`agent.cue`	`{"cue": "warmth", "intensity"?: 0.6, "at_seconds"?: 1.2}`	An expression from `at_seconds` into the current or next utterance. See Expressions.
`stage.camera`	`{"camera": "far\|mid\|close\|auto", "hold"?: true, "at_seconds"?: 2.0}`	Cuts the camera, now or timed into the next utterance.
`stage.backdrop`	`{"backdrop": "library", "at_seconds"?: 2.0}`	Swaps the backdrop.
`stage.get`	—	Answered with `stage.updated`.
`session.configure`	`{"sample_rate": 24000}`	Before the first audio only.
`session.keep_alive`	—	Does nothing; the session stays alive while the visitor's page is open.

Every command may carry an event_id. An utterance is named by the event_id of the message that opened it (we make one up if you don't), and every event about it carries that name as source_event_id.

Events you receive

type	When
`session.state_updated`	`{"state": "connected"}` on connect.
`agent.state_updated`	`previous_state` and `new_state`: `idle`, `listening` or `talking`.
`agent.speak_started`	The utterance's first audio left for the visitor. They hear it about 0.1 to 0.3 s later.
`agent.speak_ended`	Its last audio left.
`agent.speak_interrupted`	Cut by `agent.interrupt` (no `speak_ended` follows).
`agent.audio_buffer_cleared`	Confirms `agent.interrupt`.
`stage.updated`	`{"stage": {camera, camera_mode, backdrop, …}}` after a stage command.
`warning`	`{"warning": {"type": "audio_starvation"}}`: audio is arriving slower than it plays.
`error`	`{"error": {"type", "message"}}`: `invalid_request_error` (bad command, cue or audio), `queue_full`, `tts_error` (text could not be voiced), `stage_error`.

An exchange

Your own audio

→ {"type": "agent.speak", "event_id": "u1", "audio": "<base64 PCM>", "text": "Hi! Welcome in."}
→ {"type": "agent.speak", "audio": "<base64 PCM>"}
→ {"type": "agent.speak_end"}
← {"type": "agent.state_updated", "previous_state": "idle", "new_state": "talking"}
← {"type": "agent.speak_started", "source_event_id": "u1"}
← {"type": "agent.speak_ended", "source_event_id": "u1"}
← {"type": "agent.state_updated", "previous_state": "talking", "new_state": "idle"}

Text, voiced for you

→ {"type": "agent.speak_text", "event_id": "u2", "text": "[warmth] Lovely to see you again!", "cue": "welcome"}

Close codes

Code	Meaning
4000	Bad sample_rate.
4001	The session ended (you stopped it, or it ended on its own).
4002	Replaced by a newer connection to the same session.
4003	Bad or expired token.

Interrupt on barge-in: when your agent hears the visitor start talking, send agent.interrupt and then the new reply. Voice and lips stop together.