Skip to content

fix: report subagent token usage in _meta.quota.model_usage - #706

Open
danielkogan wants to merge 3 commits into
agentclientprotocol:mainfrom
danielkogan:fix/subagent-quota-model-usage
Open

danielkogan wants to merge 3 commits into
agentclientprotocol:mainfrom
danielkogan:fix/subagent-quota-model-usage

Conversation

@danielkogan

@danielkogan danielkogan commented Oct 29, 2026 •

Copy link
Copy Markdown

Problem

Codex reports usage separately for the root thread and each subagent. Clients estimating consumption from _meta.quota.model_usage previously received only the root thread's usage and missed subagent work, including when the client did not expose subagent sessions.

Behavior

model_usage now combines root and descendant-thread usage into one row per model. The adapter observes original app-server notifications before child events are routed or rewritten. Standard usage and quota.token_count remain root-thread only, so the sum of model_usage can exceed token_count.

The extension is returned in ACP v1 on PromptResponse._meta.quota and in ACP v2 on _meta.quota of the idle state_update that ends the turn. It includes children whether or not the client receives child sessions. Clients must not add separately reported child-session usage to this aggregate.

Root and child accounting share ThreadTokenUsage for repeated-total detection, inherited-history handling, model-output tracking, and request sums. Root prompts retain their total-difference calculation when a baseline is known and totals have not gone back. Child accounting uses request sums across parent prompts so a later total reset cannot revise usage already reported by a previous prompt. Pending child usage goes to the next prompt that starts model work; building a prompt's quota twice returns the same rows.

docs/air-extensions.md documents both protocol versions, overlapping scope, coverage, and the fact that token_count can be null while model_usage still contains child rows.

Coverage and limitations

  • This is an adapter-computed lower bound. Guardian review threads and title generation are not counted.
  • Codex 0.165.0 does not announce a subagent with thread/started. With an unknown starting total, a child's first usage update before any model output is treated as history; a first request without streamed output can therefore be missed.
  • A child's usage is attributed to its latest reported model, or the root model if no child model is reported. Model switches do not produce a historical per-request model breakdown.
  • The tree aggregate remains an AIR extension. It is not a drop-in equivalent of the separate own-agent reports proposed in the end-turn usage RFD; clients migrating must avoid combining overlapping usage.

Validation

  • npm run typecheck passes.
  • npm test passes: Test Files 144 passed | 7 skipped (151); Tests 2217 passed | 40 skipped (2257).
  • npm run test:integration: all 18 tests across 7 files pass, including standard subagent delivery and ownership in ACP v1 and v2.
  • npm run build passes.
  • Regression coverage includes repeated totals, inherited history, unknown baselines, counter resets across parent prompts, reported zero versus missing usage, and root/child aggregation under the same model. Prompt-path snapshots cover clients with and without native subagent sessions, and ACP v2 idle metadata.
  • A live run against Codex 0.165.0 spawned one subagent: root usage was 51,383 tokens and child usage was 16,756. model_usage contained one row of 68,139 = 51,383 + 16,756, while standard usage and token_count remained 51,383.

Daniil Kogan and others added 2 commits October 29, 2026 17:41
Codex reports the token usage of every thread on its own, and the usage of
the root thread does not include its subagents. A prompt reported only the
root model in model_usage, so clients that estimate cost from it missed
everything the subagents spent.

SubagentTokenUsage reads the notifications before the session routes them,
so it sees child usage with and without client subagent sessions, and adds
each child's usage to model_usage by model. token_count stays the root
thread's usage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex sends no thread/started for a subagent, so the tracker learns the
child from its subAgentActivity item with an unknown start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EugeneTheDev
EugeneTheDev self-requested a review November 2, 2026 11:03
Comment thread docs/air-extensions.md Outdated
| `is_mcp_tool_approval` | `session/request_permission` request `_meta.is_mcp_tool_approval: true` | The permission request approves an MCP tool call. |
| `steering` | `initialize` response `_meta.steering = {supported: true}` | The agent accepts `_session/steering` for a running turn; see [session steering](steering.md). |
| `quota` | `PromptResponse._meta.quota` | Token usage of the whole turn, the same counts as `PromptResponse.usage`: `token_count` and one `model_usage` entry for the current model, or `null` and `[]` when Codex reported no usage for the prompt, as for a prompt that started no turn. |
| `quota` | `PromptResponse._meta.quota` | Token usage of the whole turn. `token_count` is the usage of the session thread, the same counts as `PromptResponse.usage`. `model_usage` has one entry per model: the session thread under the current model, plus the subagents of the turn under their own models, so its entries can total more than `token_count`. A subagent that works on after the turn ends counts in the next turn. It is `null` and `[]` when Codex reported no usage for the prompt, as for a prompt that started no turn. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This row should spell out a few more things a client needs:

  • v2. It now also arrives in ACP v2, on the idle state_update that ends the turn. The row only names PromptResponse._meta.quota.
  • No adding. Subagents count whether or not the client gets subagent sessions, so a client must not add usage it gets on child sessions to model_usage.
  • Lower bound. The adapter computes it, and it misses some work: Guardian review threads, title generation, and a child's first request when no item streams before its usage update. Right now these are only in the PR description.
  • null with rows. token_count can be null while model_usage is not empty. A child that kept working after the last prompt adds a row to a prompt that was cancelled before the root thread reported any usage. The last sentence currently ties null and [] together.

Possible wording:

In ACP v2 it is on the idle state_update that ends the turn. model_usage counts the subagents whether or not the client gets subagent sessions, so a client must not add usage reported on child sessions to it. The adapter computes it from the usage of each thread, so it is a lower bound: Guardian review threads and title generation are not counted. token_count is null when the session thread reported no usage, and model_usage can still hold the subagent rows.

Comment thread src/SubagentTokenUsage.ts
}
return;
}
case "thread/tokenUsage/updated": {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SubagentTokenUsage copies the per-thread rules of PromptTokenUsage (src/TokenCount.ts) instead of reusing them. The rules match one to one:

Rule PromptTokenUsage.observe Here
An update that repeats the previous total is not a new request if (previous !== null && sameTokenCount(total, previous)) return; the same check
When the start is unknown, the first update before any model output carries the thread's history, so it is not counted if (previous === null && !this.modelOutputSeen) { this.startTotal = total; ... return; } if (previous === null && !child.modelOutputSeen) return;
A counted update adds its last summedLast += last child.used += last
Model output is detected with isModelOutputItem on item/started in CodexEventHandler.handleNotification, which calls observeModelOutput() in the item/started case, which sets modelOutputSeen

The comment below even says "as in PromptTokenUsage". These rules work around Codex quirks: Codex sends the current usage again when a request starts, and a resumed or forked thread's first update includes its history. When Codex changes one of these, someone has to remember to fix both places, and nothing will fail if only one gets fixed.

The two copies already differ in one place. With a known start, PromptTokenUsage reports the end total minus the start total, and falls back to summing last only when the start is unknown or the total went back. Here, a child started by thread/started has a known start (zero), but its last values are still summed. If that's on purpose, say why in a comment. Otherwise it's the kind of drift a shared helper would prevent.

Suggestion: move the shared part into one per-thread class in src/TokenCount.ts and build both on it, for example:

/** The usage of one thread from its `thread/tokenUsage/updated` notifications. */
export class ThreadTokenUsage {
    private modelOutputSeen = false;
    /** The `last` values of the updates that were counted. */
    summedLast: TokenCount = ZERO_TOKEN_COUNT;

    /** @param previousTotal the thread total at the start, or null when it is not known. */
    constructor(private previousTotal: TokenCount | null) {}

    observeModelOutput(): void { this.modelOutputSeen = true; }

    /** "repeat" and "history" updates are not counted. */
    observe(total: TokenCount, last: TokenCount): "repeat" | "history" | "counted" { /* the rules above */ }
}
  • PromptTokenUsage keeps only what is specific to a prompt (startTotal, latestTotal, totalWentBack) and uses a "history" result to record its start total.
  • Each ChildThread here keeps only model and the reported / reportedBeforePrompt bookkeeping, plus a ThreadTokenUsage in place of previousTotal, modelOutputSeen and used.

Comment thread src/CodexAcpServer.ts Outdated
(childId, itemId, request, turnId) => sessionState.subagents.withUserAction?.(childId, itemId, request, turnId) ?? request(),
event => this.observeSubagentWorkEnd(params.sessionId, event));
event => {
sessionState.subagentTokenUsage.observe(event);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This repeats observeSessionEvent. Call that instead so the two subscriptions can't drift. If the captured sessionState matters here, let observeSessionEvent take the state instead of looking it up.

});
}

for (const nativeSubagentSessions of [false, true]) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two cases differ only in input. Make this an it.each([false, true]) instead of a for loop.

});
}

it('counts a subagent of the session model in the row of that model', async () => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging a child into the root's row is reportModelUsage logic, and SubagentTokenUsage.test.ts has no case for it: none of its cases has a root row and a child on the same model. Move this check there as a unit case and drop this prompt-level test. That saves about 20 lines, and the two snapshot tests above already cover the prompt path.

@EugeneTheDev EugeneTheDev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR description: say what v2 clients get. The description only covers PromptResponse._meta.quota. In ACP v2, the same _meta.quota reaches the client on the idle state_update that ends the turn, because src/AcpV2Prompt.ts copies the v1 response's usage and _meta onto it. So this PR changes v2 output as well; the new case in prompt-v2.test.ts shows it. Please say so in the description, and in the docs (see the inline comment).

How this relates to the end-turn token usage RFD. Noting it explicitly for reference:

  • Standard usage stays root-thread only. That matches the current RFD draft, where per-turn usage covers only the session's own main loop and leaves out subagents, visible or not.
  • model_usage has no standard equivalent in the RFD. It is a per-turn total over the whole session tree, computed by the adapter, and work that outlives a turn goes to the next prompt. The RFD keeps tree totals only in cumulativeUsage, and only when the runtime measures them itself. Codex doesn't, so a standard field would leave them out. That's fine for an AIR extension as long as the docs define what it covers, which they mostly do. I'd keep it out of standard fields and revisit once subagents report their own usage on their idle states.
  • Once the adapter reports child usage on child sessions, as the RFD proposes, model_usage will overlap with it. That's why the doc should tell clients not to add the two.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants