Kotoba

Where to start

What Kotoba isWho she is and where she runs, in one page
InstallingOne package, two commands
First runA key, a model, her first words
The approval gateHow she asks before she acts
The two voice modesLocal voice, or the agent tunnel
The soul fileChange who she is
RoadmapWhat grows next, and what was cut on purpose

Or a section

↑↓ move↵ openesc close124 pages
All pages

Voice and avatar2 of 9

The two voice modes

About 4 minutes to read

On this page

voice_mode is one setting with two values, local and agent. The default is local.

api/src/kotoba/core/app_settings.py
"voice_mode": ("KOTOBA_VOICE_MODE", "local", _one_of("agent", "local"))

Change it in Settings → Personality → Voice mode, which is labelled applies to the next call, or with KOTOBA_VOICE_MODE. Both modes need an ElevenLabs key.

local — the default

Your browser opens a WebSocket to your own backend. Your backend talks to ElevenLabs.

browser ⇄ ws://<backend>/api/voice/<session_id> ⇄ backend ──outbound──> ElevenLabs

Microphone audio goes up the socket as raw PCM. The backend feeds it to ElevenLabs realtime speech-to-text, runs the turn, and streams synthesized audio back down the same socket.

What it needs from you: an ElevenLabs key. That is the whole list. Nothing has to be reachable from the internet, there is no agent to create, and the ElevenLabs Agent ID field in Settings is unused.

agent — ElevenLabs Conversational AI

An agent that lives in your ElevenLabs account holds the call. It does the listening and the speaking, and for the thinking it calls back into your backend as a Custom LLM.

browser ⇄ ElevenLabs agent (WebRTC) ──> https://<your-public-backend>/v1/chat/completions

What it needs from you, per the in-app guide in components/panels/SettingsPanel.tsx:

  1. An agent created at elevenlabs.io → Conversational AI → Agents.
  2. That agent's LLM set to Custom LLM, with Server URL https://<your-kotoba-backend>/v1/chat/completions and API Key = your KOTOBA_API_KEY.
  3. A public URL for your backend — a tunnel, or a deployed server. This is the real cost of the mode.
  4. The agent's voice id pasted into Settings → Personality → Voice ID.
  5. The agent id (agent_xxxxxxxx) pasted into Settings → ElevenLabs Agent ID.

If it will not connect, the two things to re-check are the agent id and whether the agent's Custom LLM URL and API key match this backend.

Why local exists at all

Because the two directions are not symmetric.

  • ElevenLabs' outbound APIs — realtime speech-to-text, text-to-speech — are things your machine dials out to. A laptop behind a router can do that.
  • An ElevenLabs agent in custom-LLM mode dials in. It has to reach https://…/v1/chat/completions from ElevenLabs' network, and it cannot reach localhost.

So agent mode requires a public address and local mode does not. That is the entire reason the local path was written, and it is stated at the top of core/voice/config.py:

All traffic is OUTBOUND (local backend -> EL), which is what removes the public-URL/tunnel requirement of the old Conversational-AI agent.

voice_mode is intent, not fact

Worth knowing before you read anything else in the code: voice_mode says what you meant to set up. It does not say who is on the other end of a given turn.

/v1/chat/completions is guarded by its own bearer and nothing else. If your tunnel is up, an ElevenLabs agent turn arrives there whether voice_mode says local or agent. So the rule in the code is blunt, and it is written into the tool context itself: nothing may relax a limit on voice_mode — that would relax it for exactly the caller the limit was written for.

The fact travels per turn instead. /v1's producer is the only code that knows an agent is on the other end, and it marks its own task; a ContextVar in api/src/kotoba/core/transport.py carries the mark down through everything that turn spawns.

voice_mode is still read in two places. core/loop.py uses it to decide whether a tool run gets a wordless filler hum, and — the one that matters — app_settings.audio_tags_enabled() consults it when deciding whether audio tags are asked for at all. Both are choices about how she sounds, not limits.

One consequence you can see: background work gets a different default time budget depending on who holds the call. core/work_runner.py picks it from the mark, not from the setting:

default = WORK_TIMEOUT_EL_BOUND_SECONDS (1500 s)  if an ElevenLabs agent holds this turn
          WORK_TIMEOUT_SECONDS          (3600 s)  otherwise

1500 sits under EL_MAX_DURATION_SECONDS (1800 s), which the code records as ElevenLabs' own cap on one agent conversation, with headroom left for her to announce the result before the call dies. Off that transport — local mode, or the terminal, which is neither voice mode — nothing external is holding a clock, so the real bound is the iteration and tool-call caps and the wall clock is only a backstop. The mark is captured when the job is created, because the runner is detached and the setting stays changeable while it runs. work_timeout in Settings overrides either default.

The /v1 bearer

api/src/kotoba/server.py
if not api_key and not web_pw:
    return          # the endpoint is open

/v1/chat/completions accepts KOTOBA_API_KEY or KOTOBA_WEB_PASSWORD as its bearer. If neither is set the endpoint is open. That is deliberate for a local install where the port is not reachable, but if you expose the backend for agent mode, set one of them before you expose it.