Expressions: how an emotion becomes a face

The fourteen

The emotion vocabulary is a contract, not a setting. Measured, and identical on both sides (lib/expressions.ts and api/src/kotoba/core/emotions.py):

neutral  happy  excited  sad  crying  angry  surprised
embarrassed  thinking  sleepy  affectionate  confused  scared  determined

Fourteen. Adding one means changing both files and soul/prompt.py's EMOTIONS that lists them.

When a face is sent

Emotions travel on their own channel: Server-Sent Events at /api/events/<session_id>, separate from the voice socket. The frame, measured:

event: emotion
data: {"emotion": "thinking"}

Nine emissions: five in core/loop.py, four in core/interaction.py.

A tool can declare its own profile as an EXPRESSIONS dict. Counted: 24 of the 41 tool modules do, and there are twelve distinct profiles among them — for example open_link is {"focus": "happy", "done": "happy", "fail": "embarrassed"} and patch is {"focus": "determined", "done": "happy", "fail": "confused"}. A tool that declares none gets the defaults above.

done is declared and never emitted

Of those 24 profiles, 22 carry a done face, and nothing reads it. _expr_for() is called from exactly two places, with "focus" and with "fail". That is deliberate, and it is the thing most likely to look like a bug.

The face for a finished turn comes from the audio tag she actually speaks. One source drives the voice, the browser and the terminal, instead of three that can disagree. Emitting a done face here would fight the tag a moment later.

fail is the asymmetry, and it has a reason: a stumble has to show mid-turn, before she has said anything about it.

When she wrote no tag

emotion_from_text() returns nothing, and the loop falls back to a small model call that picks one of the fourteen from her words — core/emotions.py, via the utility model. On any error it returns neutral.

This is why the face keeps working on the fast engine, where tags are switched off entirely. It costs an extra model call per turn and the guess is worse than hers, which is exactly what the prompt tells her:

A reply carrying no tag at all leaves the system to guess a face from your words, and its guess is worse than yours.

On the browser side

The incoming emotion is checked against the list itself, not with in. A frame carrying "constructor" once passed an in check — it is an inherited prototype key — and reached the model as a function, applying a random expression. And the face only applies while she is awake, in a live call; asleep keeps the drowsy face.

From an emotion to this model's expression

Nothing in the controller knows which model it drives. Parameter names, faces and parts all arrive in an AvatarConfig (lib/avatar-config.ts), and answers come in a fixed order:

1. A written profile for that folder name, if one exists.
2. What the model declares about itself, read off the loaded model.
3. A default that cannot be wrong for lack of information.

Two profiles ship. mao_pro is Live2D's free sample and the default choice; free1 remains because someone who owns a copy can still install it. Neither model is in the repository.

mao_pro's map, measured:

neutral exp_01 · happy exp_02 · excited exp_04 · affectionate exp_06 · embarrassed exp_06
angry exp_05 · determined exp_08 · confused exp_08 · thinking exp_08 · surprised exp_07
scared exp_07 · sad exp_03 · crying exp_03 · sleepy exp_03

Fourteen emotions over eight expression files. The names exp_01…exp_08 say nothing, so the map is explicit rather than guessed. The source records what each file sets — exp_02 smiling eyes closed, exp_05 brows angled down, exp_06 cheek plus blush — and calls the mapping provisional until seen on screen. Nobody rendered it to write this page either, so treat it as a starting point for your own model and remap freely.

The silent failure this guards against

pixi-live2d-display resolves an unknown expression name to false. No exception, no log — the face simply never changes. Every mapped name being one the model really declares is therefore checked by a test (tests/avatar-contract.test.mjs), and it skips rather than fails when the model is not installed, because no model is committed.

Fallback chains

A model that has three faces still answers all fourteen emotions. resolveExpression walks a chain per emotion — happy → excited → affectionate → neutral, crying → sad → scared → neutral, and so on. Every chain ends at neutral.

Demanding a complete map would turn every model missing one entry into a model that does not work.

One quirk to know. neutral's own chain is empty, so if the map has no neutral the code hands back the first value in the map. Measured with a model whose faces are named happy_face, sad_face, angry_face:

neutral -> angry_face

Not a bug — the alternative is returning nothing, and returning nothing is not a blank face: the renderer skips the call and she keeps whatever she was wearing for the rest of the session. Any face beats a frozen one. But it is a good reason to write neutral into a profile explicitly.

The adaptive layer: a model nobody wrote a profile for

adaptTo() builds a working config from what the model declares.

Faces, by name. It searches each emotion's word list inside the model's own expression names, in English and Japanese — love/heart/好き for affectionate, blush/shy/照れ for embarrassed, 怒 for angry. Measured on ['happy_face','sad_face','angry_face']:

{"angry":"angry_face","happy":"happy_face","sad":"sad_face"}

Faces, when names say nothing. If fewer than three names matched a feeling, the guess is dropped and whatever exists is spread over the emotions in a fixed order. Measured on ['exp_01','exp_02','exp_03']:

{"neutral":"exp_01","happy":"exp_02","sad":"exp_03"}

A wrong-but-stable face reads as a personality. A random one reads as a bug — which is why the order is fixed.

The mouth. From the model's LipSync group, but not blindly. The ids come out in whatever order the rigger typed them, so a parameter that actually opens the mouth wins over one that shapes it. Measured:

['ParamMouthForm','ParamMouthOpenY']  ->  ParamMouthOpenY   (not the width)
['ParamI','ParamA','ParamU']          ->  ParamA            (not the first vowel)
[]                                    ->  ParamMouthOpenY   (the default)

The blink. autoBlink is true when the model declares a non-empty EyeBlink group. It reads "the engine would blink this itself", which is why the controller takes that blinker away — see Lip sync and framing.

Things in the way. Parts whose id begins Part followed by hat, cap, hood, mask, wand, stick, prop or item, or whose author-given name contains one of those as a whole word or a CJK equivalent. Measured:

['PartHat','PartHoodie','PartHood','Part07','PartWandA']  with Part07 named 帽子沿
   ->  ['PartHat','PartHood','Part07','PartWandA']

PartHoodie is a jumper and stays on; PartHood is over her face and comes off. The word has to end where it ends. Author names come from the model's own DisplayInfo file, because most models number their parts PartN, which says nothing, and the author's name is the only place the hat is called a hat.

This guessing only ever fills what a profile did not already write. Someone who brings their own model owns that choice.

Remapping it yourself

The map lives in lib/avatar-config.ts, keyed by folder name. Add a PROFILES entry for your folder with your expressions, your mouthParam, your hideParts. From a clone running npm run dev the change is live; to keep it in an install, rebuild the packaged UI with python scripts/build_web.py, which the script's own docstring says must be run "from a clone with Node installed".

Which means, plainly: there is no interface for remapping expressions, and a pip install with no Node and no clone cannot do it. The environment variables that once overrode framing (NEXT_PUBLIC_LIVE2D_SCALE, NEXT_PUBLIC_LIVE2D_ANCHOR_Y, and the rest of NEXT_PUBLIC_LIVE2D_*) are forced to "" in the static export on purpose — a value frozen at build time would ship one machine's answer to everybody — so they are not a way round it either.

There is one visual control, and it is narrow: the first-run screen has a wardrobe beside her portrait that lists every part the model declares and lets you toggle them against the live drawing. It is a preview. The choice is not saved anywhere — nothing in the backend stores hidden parts, and reloading returns to what the profile and the guesser decided.