Repository navigation
fix: report subagent token usage in _meta.quota.model_usage - #706
danielkogan wants to merge 3 commits into
Conversation
Codex reports the token usage of every thread on its own, and the usage of the root thread does not include its subagents. A prompt reported only the root model in model_usage, so clients that estimate cost from it missed everything the subagents spent. SubagentTokenUsage reads the notifications before the session routes them, so it sees child usage with and without client subagent sessions, and adds each child's usage to model_usage by model. token_count stays the root thread's usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex sends no thread/started for a subagent, so the tracker learns the child from its subAgentActivity item with an unknown start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| | `is_mcp_tool_approval` | `session/request_permission` request `_meta.is_mcp_tool_approval: true` | The permission request approves an MCP tool call. | | ||
| | `steering` | `initialize` response `_meta.steering = {supported: true}` | The agent accepts `_session/steering` for a running turn; see [session steering](steering.md). | | ||
| | `quota` | `PromptResponse._meta.quota` | Token usage of the whole turn, the same counts as `PromptResponse.usage`: `token_count` and one `model_usage` entry for the current model, or `null` and `[]` when Codex reported no usage for the prompt, as for a prompt that started no turn. | | ||
| | `quota` | `PromptResponse._meta.quota` | Token usage of the whole turn. `token_count` is the usage of the session thread, the same counts as `PromptResponse.usage`. `model_usage` has one entry per model: the session thread under the current model, plus the subagents of the turn under their own models, so its entries can total more than `token_count`. A subagent that works on after the turn ends counts in the next turn. It is `null` and `[]` when Codex reported no usage for the prompt, as for a prompt that started no turn. | |
There was a problem hiding this comment.
This row should spell out a few more things a client needs:
- v2. It now also arrives in ACP v2, on the idle
state_updatethat ends the turn. The row only namesPromptResponse._meta.quota. - No adding. Subagents count whether or not the client gets subagent sessions, so a client must not add usage it gets on child sessions to
model_usage. - Lower bound. The adapter computes it, and it misses some work: Guardian review threads, title generation, and a child's first request when no item streams before its usage update. Right now these are only in the PR description.
nullwith rows.token_countcan benullwhilemodel_usageis not empty. A child that kept working after the last prompt adds a row to a prompt that was cancelled before the root thread reported any usage. The last sentence currently tiesnulland[]together.
Possible wording:
In ACP v2 it is on the idle
state_updatethat ends the turn.model_usagecounts the subagents whether or not the client gets subagent sessions, so a client must not add usage reported on child sessions to it. The adapter computes it from the usage of each thread, so it is a lower bound: Guardian review threads and title generation are not counted.token_countisnullwhen the session thread reported no usage, andmodel_usagecan still hold the subagent rows.
| } | ||
| return; | ||
| } | ||
| case "thread/tokenUsage/updated": { |
There was a problem hiding this comment.
SubagentTokenUsage copies the per-thread rules of PromptTokenUsage (src/TokenCount.ts) instead of reusing them. The rules match one to one:
| Rule | PromptTokenUsage.observe |
Here |
|---|---|---|
| An update that repeats the previous total is not a new request | if (previous !== null && sameTokenCount(total, previous)) return; |
the same check |
| When the start is unknown, the first update before any model output carries the thread's history, so it is not counted | if (previous === null && !this.modelOutputSeen) { this.startTotal = total; ... return; } |
if (previous === null && !child.modelOutputSeen) return; |
A counted update adds its last |
summedLast += last |
child.used += last |
Model output is detected with isModelOutputItem on item/started |
in CodexEventHandler.handleNotification, which calls observeModelOutput() |
in the item/started case, which sets modelOutputSeen |
The comment below even says "as in PromptTokenUsage". These rules work around Codex quirks: Codex sends the current usage again when a request starts, and a resumed or forked thread's first update includes its history. When Codex changes one of these, someone has to remember to fix both places, and nothing will fail if only one gets fixed.
The two copies already differ in one place. With a known start, PromptTokenUsage reports the end total minus the start total, and falls back to summing last only when the start is unknown or the total went back. Here, a child started by thread/started has a known start (zero), but its last values are still summed. If that's on purpose, say why in a comment. Otherwise it's the kind of drift a shared helper would prevent.
Suggestion: move the shared part into one per-thread class in src/TokenCount.ts and build both on it, for example:
/** The usage of one thread from its `thread/tokenUsage/updated` notifications. */
export class ThreadTokenUsage {
private modelOutputSeen = false;
/** The `last` values of the updates that were counted. */
summedLast: TokenCount = ZERO_TOKEN_COUNT;
/** @param previousTotal the thread total at the start, or null when it is not known. */
constructor(private previousTotal: TokenCount | null) {}
observeModelOutput(): void { this.modelOutputSeen = true; }
/** "repeat" and "history" updates are not counted. */
observe(total: TokenCount, last: TokenCount): "repeat" | "history" | "counted" { /* the rules above */ }
}PromptTokenUsagekeeps only what is specific to a prompt (startTotal,latestTotal,totalWentBack) and uses a"history"result to record its start total.- Each
ChildThreadhere keeps onlymodeland thereported/reportedBeforePromptbookkeeping, plus aThreadTokenUsagein place ofpreviousTotal,modelOutputSeenandused.
| (childId, itemId, request, turnId) => sessionState.subagents.withUserAction?.(childId, itemId, request, turnId) ?? request(), | ||
| event => this.observeSubagentWorkEnd(params.sessionId, event)); | ||
| event => { | ||
| sessionState.subagentTokenUsage.observe(event); |
There was a problem hiding this comment.
This repeats observeSessionEvent. Call that instead so the two subscriptions can't drift. If the captured sessionState matters here, let observeSessionEvent take the state instead of looking it up.
| }); | ||
| } | ||
|
|
||
| for (const nativeSubagentSessions of [false, true]) { |
There was a problem hiding this comment.
The two cases differ only in input. Make this an it.each([false, true]) instead of a for loop.
| }); | ||
| } | ||
|
|
||
| it('counts a subagent of the session model in the row of that model', async () => { |
There was a problem hiding this comment.
Merging a child into the root's row is reportModelUsage logic, and SubagentTokenUsage.test.ts has no case for it: none of its cases has a root row and a child on the same model. Move this check there as a unit case and drop this prompt-level test. That saves about 20 lines, and the two snapshot tests above already cover the prompt path.
EugeneTheDev
left a comment
There was a problem hiding this comment.
PR description: say what v2 clients get. The description only covers PromptResponse._meta.quota. In ACP v2, the same _meta.quota reaches the client on the idle state_update that ends the turn, because src/AcpV2Prompt.ts copies the v1 response's usage and _meta onto it. So this PR changes v2 output as well; the new case in prompt-v2.test.ts shows it. Please say so in the description, and in the docs (see the inline comment).
How this relates to the end-turn token usage RFD. Noting it explicitly for reference:
- Standard
usagestays root-thread only. That matches the current RFD draft, where per-turn usage covers only the session's own main loop and leaves out subagents, visible or not. model_usagehas no standard equivalent in the RFD. It is a per-turn total over the whole session tree, computed by the adapter, and work that outlives a turn goes to the next prompt. The RFD keeps tree totals only incumulativeUsage, and only when the runtime measures them itself. Codex doesn't, so a standard field would leave them out. That's fine for an AIR extension as long as the docs define what it covers, which they mostly do. I'd keep it out of standard fields and revisit once subagents report their own usage on their idle states.- Once the adapter reports child usage on child sessions, as the RFD proposes,
model_usagewill overlap with it. That's why the doc should tell clients not to add the two.
Problem
Codex reports usage separately for the root thread and each subagent. Clients estimating consumption from
_meta.quota.model_usagepreviously received only the root thread's usage and missed subagent work, including when the client did not expose subagent sessions.Behavior
model_usagenow combines root and descendant-thread usage into one row per model. The adapter observes original app-server notifications before child events are routed or rewritten. Standardusageandquota.token_countremain root-thread only, so the sum ofmodel_usagecan exceedtoken_count.The extension is returned in ACP v1 on
PromptResponse._meta.quotaand in ACP v2 on_meta.quotaof the idlestate_updatethat ends the turn. It includes children whether or not the client receives child sessions. Clients must not add separately reported child-session usage to this aggregate.Root and child accounting share
ThreadTokenUsagefor repeated-total detection, inherited-history handling, model-output tracking, and request sums. Root prompts retain their total-difference calculation when a baseline is known and totals have not gone back. Child accounting uses request sums across parent prompts so a later total reset cannot revise usage already reported by a previous prompt. Pending child usage goes to the next prompt that starts model work; building a prompt's quota twice returns the same rows.docs/air-extensions.mddocuments both protocol versions, overlapping scope, coverage, and the fact thattoken_countcan benullwhilemodel_usagestill contains child rows.Coverage and limitations
thread/started. With an unknown starting total, a child's first usage update before any model output is treated as history; a first request without streamed output can therefore be missed.Validation
npm run typecheckpasses.npm testpasses: Test Files 144 passed | 7 skipped (151); Tests 2217 passed | 40 skipped (2257).npm run test:integration: all 18 tests across 7 files pass, including standard subagent delivery and ownership in ACP v1 and v2.npm run buildpasses.model_usagecontained one row of 68,139 = 51,383 + 16,756, while standardusageandtoken_countremained 51,383.