The voice socket

In local mode the whole call is one WebSocket:

ws://<backend>/api/voice/<session_id>

Binary frames are audio. Text frames are JSON control messages. It is full duplex — microphone PCM goes up while synthesized PCM comes down.

It never goes through the Next proxy, and that is deliberate

By default Kotoba serves the browser app and the API on one origin: next.config.ts rewrites /api/* to the backend server-side, so there is no CORS and no cross-site cookie.

WebSocket upgrades do not survive a Next rewrite. So the voice socket, and only the voice socket, dials the backend directly. Measured on tokenUrl():

tokenUrl("/api/voice/s1")   ->  ws://127.0.0.1:8000/api/voice/s1?token=…
tokenUrl("/api/events/s1")  ->  /api/events/s1?token=…            (stays relative → proxied)

If you find this and think it is an inconsistency to clean up: it is not. Routing it through the proxy breaks voice entirely. The rule is stated in three places in the source (next.config.ts, lib/api.ts, server.py:_ws_origin_allowed).

In the packaged build the ws base is deliberately empty, so the socket dials the page's own origin. A build-time absolute URL would freeze one machine's address into everybody's copy.

Two guards on the handshake

Origin. A WebSocket handshake is exempt from CORS, so the allowlist that protects every HTTP route does not protect this one. Without a check, any page you visit could open a socket to your loopback backend and drive the agentic loop with host tools. _ws_origin_allowed measured:

origin http://localhost:3000, host 127.0.0.1:8000  -> allowed  (the dev frontend, in CORS_ORIGINS)
origin http://evil.example,   host 127.0.0.1:8000  -> refused
origin absent,                host 127.0.0.1:8000  -> allowed  (not a browser: the CLI, a script)

Default cors_origins() is ['http://localhost:3000', 'http://127.0.0.1:3000']. Same-origin is accepted too, but only on loopback — matching Origin against Host alone would accept a DNS-rebound domain, and a default install has no password to fall back on. A refused handshake closes with 4403.

Token. Once KOTOBA_WEB_PASSWORD is set, the socket needs it. A browser cannot set headers on a WebSocket, so the token rides as ?token=; the route also accepts an Authorization: Bearer header, which is what a non-browser client would use. A refused token closes with 4401. With no password set, the socket is open — the same posture as the rest of a local install.

If the socket is refused before it ever opens, the app says so rather than silently doing nothing, because typing rides the same connection:

The voice connection was refused before it opened, and typing rides the same connection, so she cannot be reached from this page at all. … if you are serving this on a different port, add that address to CORS_ORIGINS and restart.

The message contract

Enumerated from api/src/kotoba/core/voice/session.py.

Browser → backend: binary PCM frames, plus four control types.

Backend → browser: binary PCM audio, plus ten frame types — ready, partial, committed, assistant_text, audio_start, audio_end, interrupted, turn_end, skipped, error.

ready is authoritative for both audio formats. Measured:

{"type": "ready", "session_id": "...",
 "audio_in":  {"format": "pcm_16000", "sample_rate": 16000, "encoding": "s16le"},
 "audio_out": {"format": "pcm_24000", "sample_rate": 24000, "encoding": "s16le"}}

Emotion and work-mode events do not ride this socket. They come over Server-Sent Events at /api/events/<session_id>, as a separate channel. See Expressions.

The microphone

public/worklets/mic-capture.js is an AudioWorklet that resamples the device's rate (usually 44.1 or 48 kHz) down to 16 kHz s16le mono with linear interpolation, and posts frames of ~250 ms. Plain JavaScript, loaded straight from /worklets/ — no bundler.

Speech-to-text is scribe_v2_realtime with commit_strategy=vad, so the server decides when your turn ended. Measured URL parameters:

vad_silence_threshold_secs=1.5   vad_threshold=0.4
min_speech_duration_ms=100       min_silence_duration_ms=100
filter_background_audio=true

filter_background_audio is on by default. Without it an idle room commits its own noise, and every commit buys a full agentic turn plus a memory extraction. ElevenLabs rejects it alongside word timestamps, so it is dropped rather than hoped for when timings are asked for.

The transcription language is pinned in this order: KOTOBA_STT_LANGUAGE, then her configured language, then ElevenLabs auto-detection. Measured:

soul=auto, no env        -> ""   (auto-detect)
soul=es,   no env        -> "es"
soul=es,   env=ja        -> "ja"
soul=es,   env=nonsense  -> "es" + a warning; junk is refused, never sent
soul=klingon, no env     -> ""   + a warning

That second step matters. Settings → Language used to move only how she replied, so picking Spanish did nothing about a short Spanish sentence being heard as Portuguese — and the transcript is all she gets, so she answered in Portuguese. A control that cannot fix the thing it is named after is worse than no control.

The gate, and barging in

While she is speaking, your microphone frames are held back rather than sent. Otherwise her own voice comes back through the mic and the transcriber commits it as your turn.

Constants, read from lib/voice-gate.ts:

GATE_TAIL_MS 350      GATED_FRAME_BUFFER 3     VAD_WINDOW_MS 50
VAD_SUSTAIN_WINDOWS 4 VAD_MIN_RMS 350          VAD_FLOOR_RATIO 3
VAD_FLOOR_INITIAL 120 VAD_FLOOR_MIN 50

The gate closes at audio_start and reopens when local playback truly drains — the server's audio_end outruns realtime — plus 350 ms. While it is shut, frames still feed a local voice detector so its noise floor stays calibrated. The detector fires only on sustained sound: four consecutive 50 ms windows over a threshold that is the greater of 350 RMS and three times a rolling floor. A cough or a keyboard clack does not trip it. A fixed bar sat above much real post-automatic-gain speech, and barge-in never fired at all — hence the ratio.

Measured behaviour, feeding 250 ms frames to a MicUplink:

The replayed frames are the ~750 ms pre-roll ring, so the start of your sentence is not lost. The interrupt control frame must go out before them, or the backend cannot drop the stragglers of the turn it just killed.

A held bracket is different. The server opens one with no voice behind it to shut the mic while an approval card waits on a human. Its ring is dropped rather than replayed, because those frames are exactly the words the hold exists to keep off the wire.

Two limits that will surprise you

The mic reopens every 8 seconds of silence. A TTS stream nobody feeds dies on its own, and tool runs outlast it, so speech is synthesized in segments. SEGMENT_IDLE_SECS = 8.0 counts from the last text that reached the engine, not from queue traffic. Closing a segment emits audio_end, which reopens your microphone — so that budget is also how long the mic stays shut during a tool run.

Twelve voice turns per minute, then the mic goes off. VOICE_TURN_BURST = 12 in a 60-second window. The voice-activity detector needs 1.5 s of silence per commit, so no person sustains one turn every five seconds for a minute; what does is a television, a fan, or a call nobody hung up. Over-cap commits are refused (skipped, too_many_turns) and decay on their own. If they keep arriving for two whole windows running, the mic is off for the rest of the call with a message that names the likely cause — every one of those turns costs money.

Mute and unmute clears the latch. Muting also cuts audio before any speech-to-text session is opened, so a muted room is never transcribed by anyone.

There is a third, quieter one: if three transcription sessions in a row each die within five seconds of opening, the mic is disabled for the rest of the call (stt_closed). A session that lived longer resets the count. Typing still works.