Repository navigation
feat: token diet wave 4A — audit tool, subagent 5m cache, Explore + Evaluate routing, lighter agent startup, preload cleanup (#443–#447) - #457
Merged
Conversation
Add scripts/token-audit, a dev-only (not shipped) tool that reads Claude Code transcripts and reports where token spend goes: main vs subagent cost-equivalent, price-weighted by model family (unknown families are counted as unpriced), per agent type, per workflow run, spawn-length bins, cache TTL split and cache-miss events, and main-session skill loads. Run it with `npm run audit:tokens -- --project <cwd>`. Transcripts are treated as untrusted: anchored file names, no symlink following, caps on entries, lines, line length and file size that are reported when hit, and sanitized printed text. Requests are deduplicated by message id + request id (last chunk wins, output takes the max). The script tree, the new test and the integration helpers are now in the tsconfig.scripts.json typecheck.
buildSubagentsPath replaced only "/" when naming the transcript directory, but Claude Code replaces every character outside [A-Za-z0-9] with "-", so a cwd containing "." or "_" resolved to a directory that does not exist. Use the shared encodeProjectDir from the token-audit discovery module and cover a dotted cwd in the helper test.
Piping the report into `head` closed stdout mid-write and surfaced an unhandled EPIPE stack trace. Treat EPIPE on stdout as a normal end of output.
Add the subagent-cache-5m flag, default ON, which writes subagentPromptCacheTtl: "5m" to settings.json. The main session keeps the 1-hour TTL of prompt-caching-1h. Why: 99.998% of subagent cache writes were 1-hour writes, yet under 0.1% of subagent requests idle past 5 minutes, so the premium buys nothing. Modelled effect: -8.9% subagent cache-equivalent cost, -7.7% total. Probed on Claude Code 2.1.296: with ENABLE_PROMPT_CACHING_1H=true and the setting at "5m", subagents wrote only 5-minute cache while the main session wrote 1-hour. Per-agent experimental.cacheTtl was rejected in favour of one managed flag. - settingDeleteGuard now takes the onPayload type, so a scalar payload can be guarded; the comparison stays in settingValueHoldsManagedShape. Disable and uninstall delete only "5m" and leave a user's own "1h". - init seeds a guarded setting flag OFF when it is absent from the manifest and the settings already hold a value that is not its managed shape, so re-init never overwrites a user's "1h". --reset still overwrites. The rule is generic over guarded setting flags; settingHoldsUserValue is its single predicate. Refs #443
Run against real transcripts the audit tool misreported five things: - Subagents whose requests all fall outside a --since/--until window were counted as "subagents with no requests" under Skipped (1,304 in a real run). They now show on their own "Filtered by the window" line and in a new `filtered.subagentsOutsideWindow` field; `skipped.emptySubagents` counts only transcripts with no request at all. - Subagents spawned by the Workflow tool carry no toolUseId, so all of them landed in an unattributed run with no main turns (72 spawns, 4.2% of ce). Each workflows/wf_* directory is now joined to the main session's Workflow tool_use whose tool_result reports that run id in toolUseResult.runId, and so to the segment holding that tool_use. The id must match one shared wf_ pattern on both sides and is only compared, never built into a path; anything unlinked stays unattributed. - Shares and ratios carried float noise (0.9999999999999981). They are rounded to 6 decimals where they enter a report row, never in the aggregation. - workflows.byLabel summed already-rounded run figures, so a label could differ by one from the totals. It now sums unrounded values and rounds once. - The markdown report cut the directory path with an ellipsis. It is now printed whole, still control-character stripped and pipe-escaped. Each fix has a unit test that fails without it.
Ledger IDs are machine-local and may not appear in committed text. Refs #443
Fixture-only: the snapshots gain the subagentPromptCacheTtl: "5m" setting and the subagent-cache-5m manifest flag. Refs #443
The flags TUI built its rows from the manifest record alone, so a guarded setting flag the manifest had not recorded yet showed its registry default (ON). Saving then wrote the managed "5m" over a user's own "1h" under subagentPromptCacheTtl, reachable between an upgrade and the first init. handleBare now passes the record and the settings snapshot it already loaded through respectUserOwnedSettings, the rule init applies when it adopts flags, so an unrecorded guarded flag whose key holds a user value starts OFF. A recorded flag keeps its value and the managed shape still shows ON. The reducer and renderer stay pure; the settings are read in the command layer. Closes #443
Evaluate shipped Opus with no recorded rationale. The subagent baseline measured 44 Evaluate spawns, all on Opus, 3.1% of subagent cache-equivalent tokens. The alignment check compares a diff against a plan and a request, which fits the Sonnet tier Code and Test already ship; effort stays medium. The shipped row, the frontmatter line and the dynamic roster table move together. A saved `devflow agents` override still wins over the shipped tier. Refs #444
The built-in Explore inherits the session model: the baseline measured 81 spawns, 98% on Opus, at a median 11.9K tokens of starting context. devflow now ships an agent named Explore (Sonnet, high effort, Read and Bash only, no CLAUDE.md or rules), a rewrite of the built-in prompt that keeps its read-only contract, its three breadth levels, absolute paths and parallel search, and closes with the report cap. Claude Code resolves agents by exact name with user agents replacing built-ins, so the spawn sites stay as they are; plan's two Explore spawns now carry the literal type so the spawn guard sees them. The install never beside-installs over the user's own Explore: when another file under ~/.claude/agents declares the type, devflow's copy is skipped with a warning, an earlier devflow copy is removed, and the user's file is never touched. Matching is by exact name, as Claude Code's is. Registered in the plan, debug, explore and ambient plugins. Roster floors rise (agents 17 to 18, report-cap roster 15 to 16). A saved override for explore or evaluate still wins over the shipped Sonnet after a re-init. Refs #444
The open-ended "delegate all of it" left the orchestrator to improvise an agent type for work no routing line covered, and it reached for general-purpose, which inherits the session model: 93 spawns in the baseline, 83% on Opus. The routing list now ends with a fallback to the closest roster agent. It names no model, and the charter stays under its character ceiling (2699 to 2749 of 3072). Refs #444
Fixture-only: the installed agents directory gains explore.md in all three configs; the hook matrix is unchanged. Refs #444
omitClaudeMd drops both CLAUDE.md and ~/.claude/rules/**, which is about 7K tokens of every subagent spawn in this repository, billed again on each turn. Validate, Git, Triage, Evaluate, Test, Research, Knowledge, Tracker and Learning now ship the flag (Skim, Synthesize and Explore already did). Code, Scrutinize and Simplify keep CLAUDE.md. The rules those agents lose are replaced by a one-source digest: each omitting agent body carries exactly the CORE_RULES_DIGEST lines its table row declares, as bullets under a "Core rules" heading, and every digest line must still restate text in a rule file. A guard derives the omitting set from the rows and holds every body to its declaration. Learning's Encoded check compared lessons with CLAUDE.md and the rule files through the injected context, so the body now reads <project root>/CLAUDE.md and the installed rule files once, before its first already-encoded comparison. Git funds its digest block with cuts in unsampled prose and stays inside its byte ceilings.
Code runs the security scan itself: only when the session offers a code-scanning tool, once per changed file by absolute path before its final commit, one scan of the narrowest common directory past ten files, and "Security scan: unavailable" when there is no tool. Scrutinize and Simplify deny every MCP tool and say scanning is Code's job, so a scanner's tool set no longer loads into each of their spawns. mcp__* is the one wildcard a denylist may hold, and only for the agents the fixture names. The provider-scope vendor scan admits that exact line in the compiled Scrutinize agent and nothing else.
Core rules block and the cuts that fund it. Re-sets GIT_MD_CHARS (35,476), GIT_MD_LINES (800), TOTAL_CHARS (43,343), TOTAL_LINES (1,053) and GIT_AGENT_BYTES (35,703) in the same commit.
The segmentation fixture used a synthetic skill name the skill-reference guard reads as a reference to a nonexistent skill. A real command that opens no run keeps the test's intent: a non-workflow command mention must not start a segment.
Add stampAgentTools, readStampedTools and carryStampedTools beside the frontmatter rewriter. They edit one kind of entry on the inline tools line, a whole-server entry (mcp__<server>, which grants every tool the server exposes), and nothing else: other tools, other keys, the body and the file's line endings stay byte for byte. Stamping is idempotent and a new list replaces the old one. A malformed entry, a block-list tools key or a file with no tools line is refused, never repaired. The rewriter stays confined to model and effort, so a mapping reapply can never touch an agent's tool policy.
Git and Tracker now declare the built-in tools their bodies direct instead of denying a list, so a spawn is offered only those tool schemas. Git holds Bash, Read, Write and ToolSearch. Tracker holds Bash, Read and ToolSearch: its one file is written by a single scrub-gated Bash chain, so a Write tool would only be a second way to write outside it. Neither holds Grep or Glob, which Claude Code does not offer to an agent that has Bash. Neither names a tracker server: those are configured per machine and are stamped onto the installed tools line by the install. The guards are restated, not dropped. The provider-scope guard now holds the Git allowlist to the tools its bodies and references direct (a tool a body directs that the list lacks fails the build), and keeps the probe that seeds a tools key. The Tracker guard states the allowlist rationale in the agent itself and probes for a seeded server name. Tracker moves from the exemption map into the agent table with its model pinned to the session-start hook that spawns it, so the exemption map and its separate digest declaration are gone: every registered agent has a row.
… agents The Git and Tracker allowlists name no tracker server, because the servers a user has connected differ per machine. A machine setting, manifest.features.tracker.servers, lists them (absent means a default of six names) and the install appends one whole-server entry per name to the installed tools line of those two agents. devflow tracker --servers <names|default> sets the list and re-stamps the installed agents at once, in the order --set uses: manifest first, agents only if the manifest write landed. --status gains a Servers line. A name is accepted byte-exact (1-64 of A-Z a-z 0-9 _ -, no double underscore), at most 32 distinct; anything else is rejected with a message, never repaired. Every path that writes an installed copy either stamps or carries the stamp: the install stamps Git as it writes it, the Tracker converge stamps and compares the stamped text so a steady-state run reports unchanged, the learning-variant converge carries the installed stamp, and the mapping reapply, the proxy revert and uninstall go through the model-and-effort rewriter, which never touches the tools line. init and --set carry the list through to the manifest they write. A malformed servers value in the manifest heals to the default with a warning from the callers that have a terminal; the manifest still reads as a manifest.
Replacing git.md's 17-name disallowedTools block with a one-line tools allowlist took 251 ch out of dist/agents/git.md. Each of the five ceilings that carry it falls by exactly that, so every row keeps the headroom it had rather than gaining 251 ch of unspoken-for slack. Lowered only; the manifest entries move value and pattern together.
Knowledge loses apply-feature-knowledge (it refreshes an existing KB and now says to check each kept assertion against the code), Tracker loses boundary-validation, Triage loses security (the security-finding definition and the never-quote-a-live-secret rule sit inline), and Synthesize loses review-methodology and docs-framework (its review mode carries the three category scopes as a table, and /dynamic-tickets states its ticket slug rule itself). Plugins no longer require what nothing in their corpus reads: the docs-framework entry goes from code-review, debug, explore, research, bug-analysis and ambient, and the review-methodology chain goes from explore, research and plan. /plan keeps docs-framework by citing it as devflow:docs-framework on its design-document path line. The distinct skill-reference count stays at 44, so the closure floor is unchanged.
/code-review and /resolve now run git worktree list and count the worktrees first. The worktree-support skill and its discovery reference load only when more than one is listed; with one, that worktree is the only candidate and the Git agent's pre-flight validates its branch, so neither is read.
The shared decisions-load step told every host's main thread to follow the apply-decisions skill. Agents are the ones that consume the index, so the step now hands it only to agents whose contract declares it and has them apply it. explore, debug and research keep their own main-thread uses, stated in their own steps.
/debug's Offer Fix phase no longer loads the patterns and test-driven-development skills and edits in the main thread. It spawns the Code agent in implement mode with the root cause, the recommended fix and the compliance lens, then runs Simplify on the changed files. /debug loads no feature knowledge, so the Code spawn passes none. The debug plugin now declares the code agent. Code names boundary-validation, whose detection reference names security, which names review-methodology and its pattern skills, so the plugin requires that chain again alongside boundary-validation and dependency-research. The code-operation, compliance-lens and delivery guards pin the new spawn.
Under the github provider the Git agent uses gh and never reads the tracker tool-call contract, so a stamped github server is never used by a tracker operation. Under jira or linear, a connected GitHub server that also advertises issue creation would make capability selection ambiguous, which is a degraded run. A user who wants it can still add it with devflow tracker --servers.
Triage lost its security preload, so no resolve agent or command reads the skill directly. It stays reachable through the Code agent's boundary-validation reference and review-methodology, which the requires-closure guard enforces, so it moves from the plugin's owned skills to its requires list, the same shape devflow-debug uses. The installed skill set is unchanged.
The skill had drifted from what the commands write. It knew nothing of the per-ticket design folders /dynamic-plan creates, the handoff and evidence files, or the multi-issue and no-issue design documents, and its Iron Law forbade the seconds suffix /code-review adds on a same-minute collision. It now lists every path family, states the collision rule, and holds one slug rule: lowercase, each non-alphanumeric run as one dash, trimmed, at most 50. The commands follow it. /dynamic-tickets and /dynamic-plan stamp local time instead of UTC, /dynamic-tickets trims its initiative slug and caps it at 50 instead of 40, and /dynamic-plan passes the slug of the ticket set it plans and names its plans by the ticket slug. /implement, the dynamic handoff convention and /bug-analysis state how branch_slug is derived, with /bug-analysis reading the branch from the checkout it works in as /code-review does. Two rules stay as stated exceptions because a command matches by them: the ticket slug keeps nothing trimmed and no cap, and the wave slug is the integration branch's name after wave/. A guard holds the skill and the commands together, evaluating the workflow scripts' stamp and slug lines against the skill's own helpers.
…ment /plan and /research each name a path under .devflow/docs/ and had no rule for the slug or the timestamp in it: /plan said to build the path "from the slug" without defining it, /research asked for a kebab-case slug with no articles and no length cap. Each now loads devflow:docs-framework at the step that names the document, Phase 14 for /plan and Phase 2 for /research, so a run that stops earlier never pays for it, and builds the path by the skill's topic slug and local-time stamp. /research also takes seconds on a same-minute collision, as /code-review and /bug-analysis already do, and reuses one directory for every path of the run. The load is a step-scoped instruction, not a Load Companion Skills line, so the companion catalog's rows for the main thread stay as they are. devflow-research requires the skill again, and the preload guard now pins the genuine loads in place of the provenance wording.
The Synthesize agent's research mode writes research-summary.md through RESEARCH_BASE_DIR, but /research only described the summary path in prose and never passed the field, so the summary landed wherever the agent guessed instead of beside the Research agents' outputs. Phase 5 now spawns the agent with RESEARCH_OUTPUTS and RESEARCH_BASE_DIR, the same directory the Research agents write into, as /code-review and /bug-analysis already do for their modes. A guard holds all three Synthesize spawns to the field their mode writes through.
…is too
Both commands looked for the plan of the work under review in
design/*.md only, so the per-ticket plans /dynamic-plan writes under
design/{slug}/{ts}/ were never found. They also sorted by file name,
which for a /plan artifact orders by its leading issue ID, so a
low-numbered newer plan lost to a high-numbered older one.
Each now lists both layouts, sorts by the timestamp each carries (the
suffix before .md, or the parent folder's name), scans the 10 newest,
and reads the newest plan that carries the section it needs. A
per-ticket plan counts only when its ticket slug appears in the branch
name, so another ticket's acceptance criteria are never taken for this
branch's. A /dynamic-plan plan has no PR Description Guidance section,
which is why the newest plan that carries one is read rather than the
newest plan.
/release loaded devflow:git in Phase 1, so every turn of Phases 1-5 carried the skill and --dry-run, which halts after Phase 4, paid for it without ever committing. Phases 1-5 run only read-only git (status, tag listing, log) and spawn the Git agent, which preloads its own skills; the first main-thread git write is Phase 6 step 3, the release commit. The companion line now sits between the changelog step and the release commit, and says that a resume which skips step 3 loads it before its next main-thread git write. A test pins its position and that nothing before it runs a git command that writes.
The skill is now loaded on /plan and /research runs, so its size is paid there. The file-pattern table and the persistence table repeated what the directory tree already shows for every run directory. They now keep the rows the tree cannot carry, the design-document variants, the per-ticket plan, the markers and the handoff and evidence files, and the persistence rules read as one line per family of writers.
Move respectUserOwnedSettings from init-seed.ts into core/flags.ts, beside settingHoldsUserValue: the rule now serves both the init seed and the flags TUI, so the flags command no longer imports from init's seeding module. Name the stamped-agent count in runTrackerServers instead of a literal 2.
Hold the body half of scanning in the Code agent in the compiled prompts: the block scans only when the session offers a scanning tool, scans each changed file once by absolute path before the final commit, scans the narrowest common directory when more than 10 files changed, and reports `Security scan: unavailable` when there is no tool. The block names no scanner product or tool identifier, and Scrutinize and Simplify each carry the one line that scanning runs in Code. Each assertion has a known-bad probe over the same collector, and the block check fails on an empty read so the vendor absence check cannot pass vacuously.
The header still stated the git-agent golden at 35,476 ch / 800 L and the preloaded total at 43,343 ch / 1,053 L. Re-measured from the files, they are 35,225 ch / 783 L and 43,092 ch / 1,036 L, the values the equality baselines already pin. Comment only; the frozen fixture and the constants are untouched.
The notes on the five git.md-carrying ceilings said each row kept the headroom it had. The allowlist cut took 251 ch out of git.md, but the digest commit before it had added a net 2 ch (35,474 to 35,476, then 35,225), so each measurement fell 249 ch. Re-derived from the budget test's own output, headroom is now 78 on the git.md, GitHub and PR-host rows (from 80) and 56 on the Jira and Linear rows (from 58). Also refresh the two stale "measures today" lines for those rows. Wording only; no ceiling value changes.
…d, skill loads Document the wave-4A changes (#443 to #446) in the places a reader looks: the subagent-cache-5m flag and its adopt-off rule, devflow tracker --servers and the stamped Git/Tracker allowlists, the Explore override of Claude Code's built-in, the twelve omitClaudeMd agents with their core-rules digests, and the skills that now load at the step that reads them. Counts move to 18 agents and 32 flags, and the changelog gains its Unreleased entries.
The MCP-surface row said the Tracker agent declares no tools key; it holds a Bash, Read and ToolSearch allowlist and the installer stamps one whole-server entry per configured tracker server. README stated 47 generated reference files; the build emits 48, which the manifest-size floors pin and the CLI reference already stated.
The servers-line test compared plain text against a line whose "(default)" marker is dimmed through picocolors. picocolors enables colour when CI is set, so the assertion held on a developer machine and failed on the CI runner. Compare the text with its terminal control sequences removed.
…moved The install appends the configured tracker servers to the Git agent's installed tools line, so the file devflow init writes is no longer byte-identical to the compiled dist/agents/git.md. The tarball integration test still demanded byte equality and failed on CI, where the integration step runs; npm test does not run it. Assert the stamp is present on the installed copy and absent from the compiled one, then that removing the stamp gives the compiled artifact back byte for byte. A src-first installer or a publish built without the MDS compile still fails this test.
Owner
Author
Test Plan Evidence — 2e1e2baVerified 0/22: VERIFIED-CI 0, ATTESTED-LOCAL 0, UNVERIFIED 22, STALE 0, FAILED 0, INDETERMINATE 0
|
Owner
Author
Test Plan Evidence — 2e1e2baVerified 15/22: VERIFIED-CI 14, ATTESTED-LOCAL 1, UNVERIFIED 5, STALE 2, FAILED 0, INDETERMINATE 0
|
Owner
Author
Test Plan Evidence — 8d9c32eVerified 18/22: VERIFIED-CI 14, ATTESTED-LOCAL 4, UNVERIFIED 4, STALE 0, FAILED 0, INDETERMINATE 0
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Token Diet wave 4, wave A (epic #442, parent #417). Subagents are the largest remaining token cost, so this wave measures first, then cuts what each subagent pays at startup: a shorter prompt-cache TTL, cheaper model routing, no inherited instructions it does not use, and fewer preloaded skills.
Changes
#447 Repeatable token-audit tooling
npm run audit:tokensreads Claude Code transcripts and reports token use per agent, model, cache TTL and workflow run.scripts/token-audit/; it is not shipped to users.#443 Subagents on a 5-minute prompt-cache TTL
subagent-cache-5mflag writessubagentPromptCacheTtl: "5m", so subagents write 5-minute cache while the main thread keeps its 1-hour cache.#444 Subagent model routing
Exploreagent (Sonnet, high effort, read-only,omitClaudeMd) overrides the built-in Explore. Install is no-clobber: a user-ownedexplore.mdis left byte-identical, and a saved model override still wins after re-init.#445 Lighter agent startup
omitClaudeMdon 12 agents, each carrying a guarded core-rules digest, because omitting CLAUDE.md also drops~/.claude/rules/**(see V3).devflow tracker --serversstamps the whole-server entries into both agents; the default isatlassian,jira,claude_ai_Atlassian,linear,claude_ai_Linear. GitHub is left out because it runs throughgh.#446 Preload and skill-load cleanup
docs-frameworkis now the authority for document names, loaded by /plan and /research at the naming step.devflow-resolveno longer owns the security skill (moved torequires).RESEARCH_BASE_DIRto the Synthesize spawn; /implement definesbranch_slugand gates on it.Verifications (live probe 2026-10-11, Claude Code 2.1.296)
ENABLE_PROMPT_CACHING_1H=trueandsubagentPromptCacheTtl: "5m", subagents wrote only 5-minute cache (36,387 tokens, 0 at 1 hour) while the main thread wrote 100% at 1 hour. Precedence: force-5m env, thenCLAUDE_CODE_SUBAGENT_PROMPT_CACHE_TTL, then the setting, then per-agentexperimental.cacheTtl, thenENABLE_PROMPT_CACHING_1H.~/.claude/rules/**, so the core-rules digest ships with each affected agent.mcp__<server>entry grants that server's tools, unknown names do not block launch, and background spawns honourtools. Connector prefix ismcp__claude_ai_<Name>.Decisions and deviations
devflow-resolvesecurity moved torequires. The requires-closure count stays at 44 becausedocs-frameworkis genuinely loaded.git.mdceilings are lowered by 251 characters and no ceiling is raised. Floors rise: agent-roster 17 to 18, report-cap 15 to 16.Breaking Changes
None to commands, settings formats or flags. The shipped Explore agent replaces the built-in one machine-wide (see Risks).
Post-merge checks (real usage,
npm run audit:tokens)Risks to watch
atlassianandclaude_ai_Atlassianservers gets an ambiguous-server degraded run.Docs
All documentation changes (CLAUDE.md, README, CHANGELOG, CONTRIBUTING,
docs/**) are maintainer-approved.Related Issues
Closes #443
Closes #444
Closes #445
Closes #446
Closes #447
Refs #442 (epic; baseline comment #442 (comment))
Refs #417
Test Plan