Lip sync, and finding her face

How the mouth is driven

From the loudness of the audio that is playing. Not from phonemes.

Every animation frame, the canvas asks the live voice source for its output spectrum and writes the average into one Live2D parameter:

lib/audio.ts
export function getMouthAmplitude(source) {
  const data = source.getOutputByteFrequencyData();
  if (!data || data.length === 0) return 0;
  let sum = 0;
  for (let i = 0; i < data.length; i++) sum += data[i];
  return Math.min(1, sum / data.length / 255);
}

Measured:

That last row is the point of averaging the whole spectrum. The bin count differs by transport — ElevenLabs' WebRTC audio is 48 kHz, the local socket runs at 24 kHz — so hardcoding bin indices would give a different mouth in each mode.

getOutputByteFrequencyData(): Uint8Array is the ElevenLabs SDK's own contract. The local voice client deliberately implements the same method so one code path drives the mouth in both voice modes.

The honest limitation: this is loudness, not speech. There are no visemes, no phoneme alignment, no mouth shapes. Her mouth opens wide when the audio is loud and closes when it is quiet. It reads convincingly at conversational speed and it will not survive a close look at a sustained "oo".

The empty LipSync group

A Live2D model can declare a LipSync group naming the parameters its author intends for speech. Plenty of them declare it empty.

• free1 declares both its LipSync and its EyeBlink groups with empty Ids.
• lib/head-probe.ts records that two of the three models tested declare EyeBlink empty, so the file names no lid the model plainly has.

So the parameter cannot be trusted to come from the file, and Kotoba connects it manually. The order is: whatever a profile wrote, then a parameter from a non-empty LipSync group that actually opens a mouth, then ParamMouthOpenY as the default.

Getting it wrong is silent. Writing a parameter the model does not drive moves nothing at all, and writing one the model drives itself puts two writers on one mouth. That pairing — a profile's mouthParam against its model's declared group — is what tests/avatar-contract.test.mjs checks.

mao_pro is the counter-example that makes this matter: it populates its own LipSync with ParamA and its own EyeBlink. Driving it the way free1 is driven would write a parameter it does not have, and the mouth would never open.

Three rules the controller obeys

Each was measured on a loaded model when the code was written, and the numbers below are the ones recorded in lib/live2d-controller.ts. Each looks like an odd choice until you know why.

1. Nothing may write a parameter between frames. Cubism restores every parameter at the top of its own update. An outside write survives one partial frame and is then wiped — the source records 0.9 → 0.562 → 0. So all writing happens inside the frame, from two hooks.

2. Two write points, because the engine's own steps sit between them. The head goes in on afterMotionUpdate, before physics, or the hair never answers it. The face — lids and mouth — goes in last, after the expression that just applied.

3. The engine is a second animator, and it is taken away. The auto-played idle motion goes, and the model's own blinker goes with it when the model declares one. Two blinkers fight over one lid, and the engine's runs after expressions and wipes whatever eyelid one just set.

There is a trap inside rule 3. The idle motion group is pointed at a name no model can define:

const NO_MOTION_GROUP = "__kotoba_no_idle__";

"" is not safe. Live2D's own sample files its six ungrouped motions under exactly that name, so pointing idle at "" plays all six instead of none.

Two more things are switched off: autoInteract, so the model stops following the pointer, and the focus controller, pinned to centre every frame. autoInteract = false alone leaves the head drifting toward wherever the pointer last was, so she was aimed by the mouse and by Kotoba at once.

One thing is deliberately kept: CubismBreath. ParamBreath is never written. The engine drives it between the write point and the render — the source records 0.0000 → 0.2500 on a 3.23-second cycle, read from inside physics.evaluate on two models. From outside the frame the same parameter reads a flat 0.00, which is the reading that once made it look dead. Writing on top would only fight it.

Blinking

Kotoba blinks her, because it took the engine's blinker away.

The lid value is written every frame, not only during a blink — it is 1 between blinks. Leaving the lids alone between blinks looks like deference to the expression and is a way to stay shut forever: nothing else reopens them, and an expression that blends by multiplication cannot open anything on its own. Writing the base every frame composes with expressions instead of fighting them.

Timing, from lib/live2d-controller.ts: a blink lasts 150 ms, down then up. The next one is 2500–6000 ms away, except 18% of the time when it is 220 ms — a double blink.

While she is asleep, or while the lids are still ramping open after a wake-up or a startle, the lids follow the pose instead of the blink.

Framing: measuring the rig instead of typing numbers

Two numbers put her face in the card: scale (how much of the frame she fills) and anchorY (where her centre sits, as a fraction of canvas height — above 1 pushes the body off-screen, which is the whole look).

Hand-setting those per model is exactly what a project that ships no model and invites any model cannot do. So lib/head-probe.ts measures them.

Find the head by perturbing ParamAngleZ and taking the box of every drawable that moved. The tilt on Z is near-rigid; the turn on X drags the neck and the collar.

Find the eyes by closing the lids, with a stricter threshold and per-vertex filtering. The source records that closing one model's eyes drags geometry into a box 3.8 times taller than the eye; keeping a partly-moved drawable's still points out of the box is what makes the eye box an eye.

Then:

scale   = 0.96 × canvasHeight / headHeight
anchorY = 0.45 + eyeCentre × scale / canvasHeight

Measured on a synthetic rig whose head spans 0.30–0.60 of a 1.0-unit canvas with eyes at 0.45:

probeFraming() -> { scale: 3.2, anchorY: 1.89 }

which is exactly 0.96 / 0.30 and 0.45 + 0.45 × 3.2.

It degrades in layers, not all at once

A written profile always wins. Only the blanks are filled.

Four traps it was built around

It measures the rig's own rest pose, never the screen. Between frames the parameter array and the rendered vertices describe different poses, so measuring off the vertices charges the head with everything the last frame moved — a head box the size of the whole figure. It came and went, because it only showed when a frame landed first.

Hidden parts are excluded. The default model wears a hat that the app then hides. Counted in, the source records its head box measuring 37% too tall and twice too wide. So the parts have to be resolved — including a fetch for the author's part names — before the measurement, which is another reason the probe reads the rest pose rather than the live one.

The direction matters, not the distance. One model's ParamEyeLOpen runs 0..2 from a default of 1, and driving it to the maximum moves zero vertices because the rig authored no keys past about 1.1. So each measurement states which way the parameter must travel, and only a rig that answers with no movement at all is asked the other way.

The pose is always put back. The restore is in a finally. Without it, an exception mid-measure left her head tilted to its maximum for the rest of the session.

The whole thing returns null rather than throwing if the model cannot be read, and the caller falls back to the fixed pair. A model nobody profiled still loads, blinks, and has a face for every emotion.