A turn, end to end

This traces one spoken turn from the browser through the ElevenLabs agent path, because that is the path with the most moving parts. Where another surface diverges, it is called out. Every step names the module that does the work.

The example: the user says "list the files in there", she calls shell, the command is gated, and she answers.

1. The request arrives

POST /v1/chat/completions → kotoba/server.py::chat_completions.

The body is models/schemas.py::ChatRequest, which is extra="allow" on purpose. ElevenLabs nests the frontend's session id somewhere inside customLlmExtraBody, and the exact nesting is SDK-version dependent, so resolve_session_id() digs through four container names before giving up and minting a fresh one.

Before anything else:

• db.ensure_session(session_id) — the sessions row must exist, because turns.session_id is a foreign key onto it.
• interaction.note_turn(session_id) — a diagnostic that warns when a new session begins a turn while another session still has an unanswered card. It swallows every exception, deliberately: its callers invoke it unguarded, and a raise here would end the turn before turn_end is emitted, leaving the client waiting forever.
• The mute check. A muted mic still produces ElevenLabs turn-taking events; the handler answers those with a skip_turn stream unless a typed message, a finished background job, or a pending reminder overrides the silence.
• The user's text is written to turns — unless it is a trigger sentinel, which is her own machinery poking her rather than a person speaking.

2. The prompt is assembled

core/context.py::load_context.

Conversation history comes from request.messages when ElevenLabs sent more than the current line, and from db.fetch_recent_turns otherwise; either way it is capped at KOTOBA_MAX_HISTORY (24). We always own the system prompt. load_context also folds in out-of-band attachments (ElevenLabs cannot forward files to a custom LLM, so uploads arrive by a separate route and are claimed here, once), her person's profile, the memory facts, the skill titles, and the connected and pending MCP servers.

One detail worth knowing because it is easy to get wrong: the prompt's capability claims are built from what this turn will actually be offered, via the same registry.schemas_for call the loop makes. A turn that withholds tools from a speaker must not be told it has them, or the refusal reads as a fault rather than a rule.

3. The producer starts, marked with its transport

Still in chat_completions:

async with turns.lock(session_id):
    await turns.supersede(session_id)
    producer = asyncio.create_task(produce())
    turns.register(session_id, producer)

core/turns.py is the arbitration: one active turn per session. ElevenLabs posts on every turn-taking event, sometimes before the user has finished speaking, so without this a second POST starts a second agentic_loop for the same session — two loops colliding on one workdir and one event queue, with the old loop's teardown emitting working off while the new one is working. supersede cancels the in-flight turn and awaits its full teardown before the new one starts.

Inside produce(), and only here, the turn is marked:

with transport.el_call_turn():
    holder["text"] = await agentic_loop(...)

/v1's producer is the only code in the tree that knows an ElevenLabs agent is on the other end. The mark is a ContextVar, and create_task copies the context, so everything the turn spawns inherits it. See The transport mark.

4. The loop sets up

core/loop.py::agentic_loop.

There is no keyword intent router. She gets her full toolset every turn, gated only by real availability through each tool's check(), and the model decides. Setup here:

• A ToolContext carrying db, session, client, mode, channel, and — read off the ContextVar at construction — el_call_bound.
• run_id, minted per loop invocation and stamped on every frame the run emits. It is the origin field: a client deciding whether a frame belongs to the turn it is awaiting or to a detached background job routes on this key, never on "is a turn running".
• ctx.workdir = workspace.resolve_workdir(session_id) — by default the file library itself, which becomes the root path_security jails every file, shell and code operation to.
• ctx.approval = ApprovalGate(...), wired with an ask function that opens a card, an audit function that writes an audit_log row, the saved family and exact grants read out of approved_commands, and workspace_root=ctx.workdir.

5. Iterating

core/loop.py::_iterate is a while over client.responses.create(stream=True). It is hand-rolled rather than built on an agents SDK: tool calls and their outputs correlate by call_id, fed back as function_call_output items on the next call, and every output item goes back — not only the ones the API refuses to run without. Dropping her own message items asked the next iteration of a model with no record of having spoken, and it wrote the same paragraphs twice.

Before each call, core/ratelimit.py::throttle may pause; see Rate limiting.

Text deltas go onto stream_queue as they arrive. When an iteration produces tool calls instead of a final answer, the loop emits a FLUSH_SENTINEL so the spoken filters release what they are holding — without it a reply ending "…the code." sits in the filters until the next iteration's text arrives, measured at 10.09 s of silence on a 10 s tool.

Caps, from core/loop.py:

The + 1 is load-bearing. The gate that drops every tool ("you are out of calls, answer in words") is evaluated at the top of an iteration, so at one call per iteration it needed iteration 41 of 40 and never ran: the chain just stopped, and the work runner announced "Done." as a success.

6. A tool runs

core/loop.py::_announce_action fires before execution and emits, in order:

1. Once per turn, for an action: emotion determined + a working on task frame.
2. The tool's focus expression (ToolSpec.expressions, default thinking).
3. Either a step frame with phase="start" (an action) or a peek frame (a read-only tool).

Then execute_with_heartbeat runs it. Dispatch goes through registry.dispatchable, which knows whether the family is switched off; an unregistered name, a withheld name and a switched-off family are three different sentences handed back to the model, and all three are ok=False. The heartbeat is 3-second ticks with the first phrase at ~9 s and one every ~21 s after — a cadence picked by ear, because four hums in twelve seconds was measured as grating. Beats are deferred, not skipped, while the tool is blocked on a human: the card already says everything.

The traced run emits exactly this sequence on the event channel:

emotion  determined                       run_id=e6bcaa16
task     working on=true
emotion  determined            (shell's focus expression)
task     step phase=start id=c1 step_kind=shell action="$ ls"
task     step phase=done  id=c1 ok=true outcome=ok result=... full=...
task     files_changed
emotion  happy                 (from the audio tag she wrote)
task     working on=false

7. The gate asks

shell is risk="exec". On the ElevenLabs transport, with a listener registered, it does not block — it schedules through core/deferred_exec.py and ends the turn saying it has asked. On every other transport it blocks inline on ctx.approval.confirm. The criterion is the transport, not the channel; that whole story is Inline versus deferred.

An approval has five endings — APPROVED, DECLINED, UNANSWERED, UNREACHABLE, DISMISSED — and core/interaction.py derives every sentence about the ending from that one word: the audit row's approver (approver_for), the terminal row's phrase (refusal_row), and what the model is told (refusal_note). A (False, False) tuple can name only one of the five, which is how a typed command that expired unread at 181 s was once filed as approved=0, approver='user' — the trail asserting the user saw it and refused it.

8. The answer comes back

Two things happen with her words, on two independent channels.

Channel A — the reply text. event_generator in server.py drains stream_queue and re-frames it as OpenAI chat.completion.chunk SSE, through a stack of filters from core/stream.py: tool-call leaks, code fences, bare URLs, audio tags, forbidden phrases. If the queue is silent for 4 seconds it emits a buffer word — ElevenLabs cuts a turn after about 7 seconds without audio, and that buffer never counts as real text.

Channel B — the face. core/events.py::emit_emotion pushes onto the session's queue, which GET /api/events/{session_id} is draining as event: emotion frames. The face for a finished turn comes from the audio tag she actually wrote (stream.TAG_TO_FACE, 31 tag spellings onto 14 faces), so voice and face have one source and cannot disagree. Tools declare a done expression and nothing reads it — emitting one here would fight the tag a moment later.

The two channels never touch, which is why lip sync and expression cannot fall out of step.

9. Teardown

agentic_loop's finally runs even on CancelledError, or a cancelled turn leaves terminal cards spinning: any still-open step is completed with outcome="interrupted", the peek line is cleared, the working chip is turned off unless a background job still owns it, finished reports are published to the Files panel, and ctx.cleanup() runs.

server.py's own finally then persists the assistant turn, clears any pending reminder, and detaches extract_and_save_memory as a background task with a strong reference held in _bg_tasks — a bare create_task can be garbage-collected while running.

The same trace on the other surfaces