Rate limiting and back-pressure

An agentic turn is token-heavy: the system prompt, the history, the tool schemas and every tool result go back on every iteration. A long work run can exhaust an organisation's tokens-per-minute ceiling. core/ratelimit.py handles it in two layers — a proactive one that reads the provider's own headers, and a reactive one behind it.

The state is process-global on purpose. TPM is an organisation-wide budget shared across every turn and every session, so a per-session budget would be describing something that does not exist.

                      response headers
                             |
                    ratelimit.note_stream / note_headers
                             |
                      _state = {remaining, reset_at}
                             |
   before each call ---> throttle()  ---> 0.0        "clear to send"
                             |         \
                             |          -> owed > 0 -> wait_out(owed) -> send
                             v
                     client.responses.create(...)
                             |
                        429 / RateLimitError
                             |
                  backoff_seconds(attempt, retry_after) -> sleep -> retry
                             (bounded ladder, 6 attempts)

Proactive: pacing on the provider's headers

note_stream(stream) reads the rate-limit headers off the AsyncStream's underlying httpx response after every call, keeping two numbers: x-ratelimit-remaining-tokens, and a monotonic deadline derived from x-ratelimit-reset-tokens. Those reset headers come in shapes like 6m0s, 1.5s, 13ms or 2m, so _parse_duration handles all of them and falls back to reading a bare number as seconds.

throttle() runs before every responses.create. If the last-seen remaining budget is below the floor, it sleeps until the window resets, plus jitter.

The two ceilings say the same thing from opposite sides. Pacing exists for a per-minute token bucket: a reset further out than _max_pace() is a header describing something else — a daily cap, a clock skew — and is not worth stalling a live turn on, so we stop pacing and let the reactive layer carry the request. Likewise a Retry-After well past a minute is a daily or monthly quota, and parking a live turn on it is worse than letting the bounded ladder fail the turn.

A wait cut short is not a wait saved

This is the part worth understanding, because both halves used to get it wrong and the symptom was more dead air, not less.

throttle takes a max_wait — 5 s on a companion turn, 65 s in work mode — which protects a live voice turn from silence. But trimming the sleep never made the request affordable. The caller sent into a near-certain 429, spent a slot from its retry ladder and backed off anyway, for several times the dead air the cap was trimming.

So throttle no longer caps and shrugs. It returns the seconds it could not wait: 0.0 means clear to send, and a positive number means the sleep was cut short and the budget is still below the floor. That return value is not advisory — core/loop.py reads it and pays the remainder:

owed = await ratelimit.throttle(max_wait=65.0 if mode == "work" else 5.0)
if owed:
    await ratelimit.wait_out(owed)

max_wait now bounds only what that one function stalls for, not what the caller ends up doing.

Reactive: the backoff behind it

If a call raises anyway, _is_rate_limit(exc) decides whether it is worth retrying. It matches the SDK's RateLimitError when available, and otherwise looks for rate-limit wording in the message — with word boundaries on the short tokens, because a bare "429" in s matched request ids and byte counts, sending an ordinary failure through six exponential backoffs (about a minute) before producing the same error.

backoff_seconds(attempt, retry_after) is exponential with jitter — base * 2^(attempt-1) ± 20%, capped at 30 s — except when the server supplied a Retry-After, which is honoured in full up to max_retry_after(). The cap bounds our guess and nothing else. Truncating a 55 s answer to 30 sent the retry back inside a window we had just been told was closed: it 429'd again, burned a slot out of the bounded ladder, and made the total wait longer than honouring the number would have been.

Two conditions end the ladder in core/loop.py: exceeding _MAX_RL_RETRIES, or the error not being a rate limit at all. There is a third guard worth noting — a retry is refused once the iteration has already produced text on a non-subagent turn (response_text and not sub), because re-running that call would speak the same words twice.

Back-pressure that is not about the provider

Three other bounds shape a turn, and they are in core/loop.py rather than here:

• Tool output caps. 16,000 characters fed back to the model per call, 24,000 for browser__* tools, whose snapshots are large. When it clips, it says so — a silent truncation is worse than a short one. The panel carries the same cap for its expander, so the two copies cannot disagree about what she was allowed to read.
• Browser history elision. Earlier browser views are replaced with a one-line placeholder telling her to take a fresh snapshot, so a long browsing run does not carry every intermediate DOM.
• History depth. KOTOBA_MAX_HISTORY (24 turns), deliberately not split by transport — the two would need the same number, and a second one is only a way to get it wrong.