Audio tags, and the face they pick

She writes square-bracket cues into her reply. eleven_v3 performs them, so [whispers] over here is actually whispered. The same tag also picks the expression her avatar wears. One thing she writes, two things it drives.

The list

Nineteen tags. Anything else is stripped before it reaches ElevenLabs. Enumerated from ALLOWED_AUDIO_TAGS in api/src/kotoba/core/stream.py:

The prompt tells her three things about them that matter:

• Placement. A tag colours the words after it, so it must lead a clause and have real words following it in the same sentence. A tag at the end of a sentence gets spoken aloud instead of felt.
• Never invent one. [yay] or [smiles] is read out literally.
• Always English. She writes her tags in English even when speaking Japanese or Spanish. A translated tag is read aloud.

The face each one picks

Measured by running emotion_from_text("[tag] hola") for all nineteen:

The map is TAG_TO_EMOTION, and it is deliberately the only copy: the voice and the face are resolved from one table so they cannot disagree.

She can also write a face name directly. TAG_TO_FACE is TAG_TO_EMOTION plus the fourteen emotion names mapped to themselves, so [thinking] and [determined] — which she writes often, and which are faces rather than voice tags — resolve too. Measured: [thinking] a ver... gives thinking.

Brackets that are not tags stay brackets. Measured:

"[Forbes](https://x.com) dice que..."  -> no face   (a markdown link)
"arr[0] vale 3"                        -> no face   (an index)
"hola que tal"                         -> no face   (no tag at all)

The trap: the face works on any engine, the voice does not

This is the one worth knowing.

There is a single gate, app_settings.audio_tags_enabled(), and both consumers — the prompt that asks her for tags, and the stream filter that keeps them — read it. Measured across every combination:

Two things follow.

expressive off wins everywhere. It is a toggle in Settings → Brain, labelled Expressive voice (audio tags). Off, the prompt stops asking for tags and the filter strips any she writes anyway.

fast disables tags in local mode, on purpose. The flash engine deletes every tag before synthesis, so asking her for them would buy prompt tokens and reply tokens for a flat voice. In agent mode the engine setting is ignored, because the performer is your dashboard agent and Kotoba cannot see which model it uses — there, expressive alone decides.

But the face keeps working either way. The face is read by a filter that does not consult the gate at all (emotion_from_text builds its own AudioTagFilter(keep_valid=False, text_surface=True)), and when a reply carries no tag the loop falls back to a small model call that picks one of the fourteen emotions from her words (api/src/kotoba/core/emotions.py).

So the failure mode is silent and asymmetric: expressive: true with tts_engine: fast means her face keeps emoting while her voice goes flat, and nothing in the interface says so. That combination ran silent for weeks before it was caught.

kotoba doctor now catches exactly this, as a warning rather than a failure, because fast is a legitimate choice:

voice tags  warn  asked for but never performed — expressive is on, but tts_engine `fast` strips
                  every audio tag before synthesis, so her face emotes while her voice stays flat.
                  Set tts_engine to `expressive` to hear them, or set expressive off if fast is
                  the point.

What happens to a tag she should not have written

Measured through the real spoken chain:

"[yay] Hola [warmly] que tal"   ->  " Hola [warmly] que tal"

[yay] is not in the list, so it is removed; [warmly] is, so it survives to the engine. The filter is streaming-safe — the model emits [, hap, pily, ] as separate tokens and it is still recognised as one tag.

On the spoken path that is the whole story: a voice must not read a bracket it cannot perform, so every bracket goes unless it is a known tag and tags are switched on.

The written surfaces need a second, narrower filter, and it runs only there — the terminal (cli/session.py) and the Discord bridge, not the browser's spoken chain. A terminal shows markdown, where most brackets are prose, so it keeps them; but the model still coins near-vocabulary tags, and one of those at the head of a reply is not prose. Position is the evidence the vocabulary cannot give: a bracket before any other character, holding only lowercase letters and not opening a markdown link, sits exactly where she was told to put her tag. So a coined [sleepyhead] is dropped from the head of a written reply, and if its words are in the map it picks the face on the way out. One printable character ends that watch for the rest of the turn, which is what keeps [1], list[int] and [Forbes](url) out of it.