Providers, models and keys

Two providers, and what "provider" means here

Kotoba ships descriptors for exactly two: openai and xai. Those are the only values provider accepts, and the only two entries in the code's provider table.

That table is small because the loop is coupled to the Responses API shape, not to a vendor. Every model call in the codebase is client.responses.create(...). So a third company is a near-drop-in when it serves that API — and is simply not reachable when it serves Chat Completions only.

Pointing base_url somewhere else

base_url takes any http:// or https:// URL, so an OpenAI-compatible gateway in front of another model works — provided it serves the Responses endpoint, streams, and calls tools. Kotoba will not stop you: serves_model gives an unknown-shaped model name the benefit of the doubt precisely so a gateway is usable, and a false "your provider does not have that model" would be worse than the 404 it saves. Nothing in the product validates a gateway beyond the round trip you can run yourself from Settings → Brain → Test connection.

The offered models

These are what first run and the Settings menus offer. Typing any other id is allowed — the list is a recommendation, not a ceiling. Prices are dollars per million output tokens as recorded in the code; the multiplier is against that provider's cheapest, and rows within 5% of it get none.

OpenAI

xAI

A model earns a place on a list by answering the Responses API, calling tools, streaming — and by reasoning, because a non-reasoning model makes Kotoba fall back to short canned narration lines that are English whatever language you are speaking.

Per-role models

Five roles, one chain each. Empty means "inherit", and the chain is walked left to right until something is non-empty:

Note the asymmetry: utility does not inherit from work_model. Set work_model and the code and research roles follow it; the utility sidecars stay on the companion model until you set utility_model yourself.

The utility role is exactly two things: emotion extraction and memory extraction, which both go through one helper. The screenshot captioner does not — it calls the companion model directly, whatever utility_model says. The Settings panel labels the control "Utility model (emotion / memory / captions)", which over-promises on that last word.

Measured, starting from an empty configuration:

Reasoning arguments are decided per role, on the model that role will actually use. A cheap non-reasoning utility_model therefore receives no reasoning block and no encrypted-content include — which is the arrangement that stops emotion and memory dying with a silent 400.

Where a key lives

There are two places, and one beats the other.

Saved in the app. kotoba setup, the browser's first-run screen and Settings → Brain all write here. The key is encrypted with AES-256-GCM and stored in the saved_keys table under the name llm:<provider>:api_key; the ElevenLabs key goes to voice:elevenlabs:api_key. At startup the engine decrypts them into process memory. They are never logged, never returned to a caller and never put in the model's context.

In the environment. OPENAI_API_KEY / XAI_API_KEY for the model, ELEVENLABS_API_KEY for the voice. Used only when nothing usable is saved in the app.

Resolution is: saved key → environment variable → no key. With no key the server still boots, the SSE pipeline still runs, and the loop answers with a friendly offline message instead of failing.

Verify, then store

Both front doors probe the candidate key with a real round trip before anything is written, on a client built for that key alone. A refused key is a 400 and nothing is saved.

The order is not cosmetic. Saving first left keys in the keystore that had never answered, and reordering the two calls could not fix it, because the other test — Settings → Brain → Test connection — validates whatever is already active, so it could only ever test the previous key.

The probe also picks the model: if the configured model is not one this provider serves, it probes with the provider's own default, because a model belonging to the other company answers 404 and reads to a person as a bad key.

For ElevenLabs the check is GET /v1/voices, deliberately not /v1/user — keys carry scopes, and a key that speaks perfectly answers /v1/user with a 401. Only two answers throw an ElevenLabs key away: ElevenLabs itself saying invalid_api_key, and a 429 against that key. A captive portal, a proxy, a timeout or a 5xx all keep the key and say so.

The mismatch that looks like a bad key

provider and model are two independent settings, and abandoning first run between them leaves a good key pointed at a model its provider does not serve. Every turn then comes back empty.

That is why the browser panel does not write them through the raw runtime endpoint. Picking a provider goes to /api/setup/provider, which pins model to something that provider serves and then persists the provider; picking a model goes to /api/setup/model, which refuses a model belonging to the other company and names who does serve it. kotoba setup runs the same two functions.

kotoba doctor reports the pair as a failure with the same sentence, rather than the green tick it used to give.

Changing provider

From the terminal, /set provider xai asks you to confirm first, saying what it means: that sends everything you say to xAI instead of openai — a different company, a different privacy policy, and the keys are kept apart. The keys really are kept apart: each provider has its own row, and switching back does not need the first key re-entered.

From the browser the Provider select applies immediately, with no confirmation step.

Reasoning state

By default Kotoba is stateless with the provider: it sends store=False together with include=["reasoning.encrypted_content"] and carries the reasoning itself, re-feeding the encrypted blob each iteration. Nothing is retained on the provider's side.

KOTOBA_LLM_STORE=1 opts into provider-stored state instead — store=True, and the include is dropped. Set it only if your model rejects encrypted reasoning.

Measured on a reasoning model with reasoning_effort: low:

companion → {"reasoning": {"effort": "low"},    "store": false, "include": ["reasoning.encrypted_content"]}
work      → {"reasoning": {"effort": "medium"}, "store": false, "include": ["reasoning.encrypted_content"]}

and on a non-reasoning model, or with reasoning_effort: off, the extra arguments are simply absent.