Emotion and the face

There are fourteen faces. Which one she wears comes from the audio tag she writes, and the same tag also tells the voice engine how to perform the line. One source for both. That design decision looks like a missing feature until you know why, so this page explains it before anything else.

The fourteen

From api/src/kotoba/core/emotions.py, mirrored exactly in lib/expressions.ts:

neutral   happy      excited   sad        crying     angry     surprised
embarrassed  thinking  sleepy   affectionate  confused  scared  determined

That list is a contract, not a setting. The TypeScript file says so:

"The Emotion union MUST equal the backend's VALID_EMOTIONS and the soul/default.md emotion list."

Three places name the fourteen — the backend, the browser union, and the list the prompt hands her — and api/tests/test_the_three_emotion_lists_agree.py compares all three.

You cannot add a fifteenth from the soul file. Which expression each of the fourteen plays on your Live2D model is configurable and lives elsewhere — lib/avatar-config.ts — because that answer changes with the model and this list does not.

The soul file's ## Emotional rules section names thirteen of them directly in an emotion: slot, and mentions crying as an escalation of sad.

The nineteen audio tags

These are the square-bracket cues the expressive voice actually performs. From core/stream.py:ALLOWED_AUDIO_TAGS — nineteen, and this is the complete list:

Anything else — an invented [yay], a translated [emocionada] — is stripped from the spoken stream and sets no face. That is enforced in code, not asked of the model.

Tag to face

core/stream.py:TAG_TO_EMOTION, in full:

TAG_TO_FACE is that table plus the fourteen emotion names mapped to themselves — 31 keys, because sad and excited appear in both lists.

That second half is the useful bit. Twelve of the fourteen emotion words are not performable tags. Writing [thinking] picks the thinking face and is deleted before the voice ever sees it. Measured:

'[thinking] Mmm...'      spoken=' Mmm...'          face='thinking'
'[determined] Vale.'     spoken=' Vale.'           face='determined'
'[warmly] Hola.'         spoken='[warmly] Hola.'   face='affectionate'

The twelve face-only words: neutral happy crying angry surprised embarrassed thinking sleepy affectionate confused scared determined.

This matters because the written register names those words for her, and matching brackets against the nineteen audio tags alone left [thinking], [determined] and [confused] — the ones she writes most — falling through to a second model call in the browser while the terminal had already resolved them.

What is emitted during a turn

Four things move the face, and only four.

An "action" is a tool whose risk is write, exec or network; a plain read does not raise the chip.

24 of the 40 registered tools declare a face profile. A few, read out of the registry:

shell         focus=determined  done=happy      fail=embarrassed
read_file     focus=thinking    done=happy      fail=sad
execute_code  focus=thinking    done=excited    fail=confused
make_report   focus=determined  done=excited    fail=embarrassed
skill_view    focus=thinking    done=determined fail=confused

Those profiles are code, in each tool's module as an EXPRESSIONS dict. The soul file cannot change them. test_personality_contract.py asserts every declared value is one of the fourteen — a typo there would silently no-op the face.

Why success emits nothing, on purpose

You will notice done in that table. Nothing reads it. Only focus and fail are ever emitted.

This is deliberate, and somebody will try to fix it. From core/loop.py:_expr_for:

"Only focus and fail are emitted. Success is deliberately silent: the face for a finished turn comes from the tag she actually wrote, which is one source for voice, browser and terminal instead of three that can disagree. Tools still DECLARE a done face — nothing reads it today, and adding an emission here would fight the tag a moment later."

Read it as a sequence and it is obvious. A tool finishes, and a fraction of a second later she starts speaking her answer, which opens with a tag. If the tool emitted happy on success and her sentence opens [sad] I'm afraid that didn't find much, the face flips twice in under a second and settles on the one that contradicts the words.

fail is the asymmetry, and it has a reason: a stumble has to show mid-turn, before she has said anything. There is nothing yet to read a tag from.

The test that pins this says it too:

"The state names are the declarable vocabulary, not the emitted set: only focus and fail reach the avatar. done is declared and unread on purpose."

So: an unread done key is not a bug, an unemitted success face is not a bug, and "restoring" the face after a tool succeeds is the change that breaks it.

When she writes no tag

emotion_from_text returns None. The loop then falls back to core/emotions.py:extract_emotion, which is a second model call asking for one word out of the fourteen. Any error returns neutral.

The prompt is explicit that this fallback is worse than her own choice, and the reason it says so is recorded in the code. The block used to tell her the face was out of her hands:

"The code says otherwise: the face is read from the tag she speaks, and inference is only the fallback for a reply that carried none. Told she did not control it, she had licence to stop writing tags, which costs the voice and the face at once."

The block that ships now is the last thing in the prompt, so it wins:

"One channel, not two: the audio tag you write picks BOTH how you sound and the face you wear. Write the tag you mean. A reply carrying no tag at all leaves the system to guess a face from your words, and its guess is worse than yours."

Whether she writes a tag is an instruction. Whether a tag she writes picks a face is code.

The two surfaces do not update at the same moment

The terminal passes an on_tag callback into the same filter (AudioTagFilter(on_tag=…)) so the face is painted with her first frame instead of repainted after her last. The browser has no such callback: it gets the tool faces mid-turn, and then one final emotion event carrying the face of the first tag in the finished reply. A second tag later in the same reply moves the terminal's face and not the browser's.

api/tests/test_one_face_map.py pins that both surfaces resolve a given bracket to the same face — they read the same filter and the same map.

An emotion event with nobody listening is dropped, silently, by design (events._put returns early when no queue is registered for the session).

Tags off

Audio tags have one gate: core/app_settings.py:audio_tags_enabled(). Both the prompt (does it ask for tags?) and the stream filter (does it keep them?) delegate to it, so they cannot disagree — they once read the environment separately and desynced, with the prompt asking for tags the filter was stripping.

Measured rule table:

kotoba doctor reports the middle row as a warning, because it ran silent for weeks:

voice tags   asked for but never performed — expressive is on, but tts_engine `fast` strips every
             audio tag before synthesis, so her face emotes while her voice stays flat. Set
             tts_engine to `expressive` to hear them, or set expressive off if fast is the point.

It is a warn and not a fail on purpose: fast is a legitimate trade — lower latency, no tags.

With tags off, the prompt swaps in a different emotion block:

"You write no tags in this mode, so your visible avatar expression is read from your words — the system picks one of: … Just respond naturally."

Which means: with tags off, every turn's face costs a second model call.

Where to put a tag

This is a prompt rule, not an enforcement, but it is the single most common way a tag goes wrong, so it is worth repeating here. A tag colours the words that come after it:

GOOD  [warmly] It's so good to hear from you.
GOOD  [excited] Wait, you got the job?! [laughs] That's amazing!
BAD   Here's the result: [sad] oh no.
BAD   That's great [excited]
BAD   [happy][giggles] hi

A tag with little or nothing after it gets spoken aloud — the voice says the word "sad" instead of sounding sad. Nothing in the code prevents that; the tag is valid, it is passed through, and the engine reads it.

Two tag words glued into one bracket are handled — the model does this, and it was seen live. Measured, on both surfaces:

Row three is the one to notice: [sad] and [warmly] are each a valid audio tag, so both are kept and handed to the voice engine back to back — which is exactly the stacking the prompt tells her not to do. The filter has no opinion about it; only the prompt does.

Row four is a deliberate refusal — a connector makes the whole bracket prose, because "classifying near-matches to the vocabulary is the mistake that once rewrote ((url))". It reaches the terminal as visible noise and reaches the voice as nothing at all.