diff --git a/docs/plans/issue-3829-session-human-touch-points.md b/docs/plans/issue-3829-session-human-touch-points.md new file mode 100644 index 000000000..89747cfd1 --- /dev/null +++ b/docs/plans/issue-3829-session-human-touch-points.md @@ -0,0 +1,196 @@ +--- +owner: migmartri +ticket: https://github.com/chainloop-dev/chainloop/issues/3829 +spec: docs/specs/issue-3829-session-human-touch-points.md +--- + + + +# Plan issue-3829: Record the human touch points of an AI coding session + +## Overview + +This plan covers milestone 1 of the spec: the evidence section and Claude Code. The Claude Code parser already counts the messages of a session at push time. It will also count the human touch points and the permission modes, and write them into a new human interaction section of the session evidence. A new hook records each permission request in the Trace state, as the skill hooks record skill loads. The push reads these records and gives them to the parser. + +## Architecture and impact + +```mermaid +flowchart LR + A[Agent] -->|transcript| C[Transcript copy] + A -->|permission request hook, Claude Code only| S[Trace state] + C --> P[Provider parser at push] + S -->|permission requests of the session| P + P -->|human interaction section| E[Session evidence] + E --> V[Schema validation] +``` + +The components stay the same. The change adds one hook path for Claude Code and one new output of the Claude Code parser. + +- Session evidence format: the new section, its schema, and its redaction rule. +- Claude Code hooks: the new permission request hook and its state record. +- Push assembly: reads the permission requests from the state and passes them to the parser. +- Claude Code parser: counts the touch points and the permission modes. +- Trace README: the new rows of the compatibility matrix. + +| Impact on | What changes | +|-----------|--------------| +| Users and other parts of the product | The session evidence gets a new section. The existing conversation counts do not change. The platform can ignore the section until a later change reads it. | +| Data and migrations | None. The evidence is a material, not a table. | +| Configuration and deployment | `trace init` installs one more Claude Code hook. A repository that ran `trace init` before gets it on the next `trace init`. Until then, its evidence has no permission counts. | +| Compatibility | The section is additive and the schema version stays the same. Only the CLI runs the schema validation, with the schema in its own binary. The control plane does not validate the evidence, so a new CLI works with an old control plane, and no release order applies. | +| Rollback | Revert the CLI. Evidence that is already stored keeps the section, and the schema still accepts it. | + +## Milestones + +| # | Milestone | Requirements | Depends on | +|---|-----------|--------------|------------| +| M1 | Evidence section and Claude Code | R-001 to R-010 | none | + +### M1: Evidence section and Claude Code + +The session evidence gets the human interaction section, and its schema accepts it. The Claude Code parser fills all the counts and the permission modes, from the transcript of the session and its subagents. A new permission request hook records each request in the Trace state, with the pattern of the skill hooks (rung 2). The README matrix gets the new rows, with the cells for Claude Code set and the other cells at **No** until their milestone. + +**Done when:** a pushed Claude Code session holds the section with all counts and its permission modes. An old session without the section still passes schema validation. The parser tests with new transcript fixtures and the schema validator tests prove both. + +**Steps**, in build order. Open a step to see what changes and the files it touches. + +
+01 / 06 · The session evidence can hold the human interaction section (R-009, R-010) + +Add the section to the evidence data with the fields of the spec Proposal table. Each count is a pointer, and the list of permission modes is a nil slice when not recorded. So a zero stays in the JSON and a missing count stays out (R-009). The JSON schema gets the new optional property, with the same open string rule as `session.mode` for the mode names. The secret redaction keeps the mode names, as it keeps `session.mode`. + +| File | Change | +|------|--------| +| `pkg/attestation/crafter/materials/aicodingsession/aicodingsession.go` | Add `StartedBy string` to `Session`, with the `started_by,omitempty` JSON tag, so an unknown starter is left out. Add `HumanInteraction` with `PermissionModes []string` and a `TouchPoints` struct of `*int` fields. Add `HumanInteraction *HumanInteraction` to `Data`. | +| `pkg/attestation/crafter/materials/aicodingsession/aicodingsession_test.go` | Add cases: a zero count stays in the JSON output, a nil count stays out, and an empty `StartedBy` gives no `started_by` key. | +| `internal/schemavalidators/internal_schemas/aicodingsession/ai-coding-session-0.1.schema.json` | Add the optional `human_interaction` property. State in its description that the mode names are native to each agent. Add the optional `session.started_by` string, open like `session.mode`, and name its values in the description. | +| `internal/schemavalidators/schemavalidators_test.go` | Add cases: evidence with the section and `started_by` validates, and evidence without them validates. | +| `pkg/attestation/crafter/materials/aicodingsession/redact.go` | Add `/data/human_interaction/permission_modes/*` and `/data/session/started_by` to the protected paths. | +| `pkg/attestation/crafter/materials/aicodingsession/redact_test.go` | Add cases: a mode name and a starter value that look like secrets stay. | + +
+ +
+02 / 06 · Claude Code sessions record prompts, questions, plans, interrupts, and permission modes (R-001, R-002, R-004, R-006, R-007, R-008) + +The parser already reads each transcript once: `scanJSONL` decodes each record into `jsonlRecord` and calls back `parseJSONL` for the main transcript and `parseSubagentUsage` for each subagent transcript. The counter adds no second read and no second loop. It is an accumulator that these two callbacks feed with each record, and `jsonlRecord` gets the fields that it reads. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-001). The permission modes come from the `permissionMode` field of the user records, in the order of first use. `scanJSONL` drops a line that it cannot decode before the callback runs. So it will also report how many lines it dropped. When it dropped a line, the parser leaves out the touch point counts and adds a warning (R-010). + +| Rule input | Count | +|------------|-------| +| User record of the main transcript that `isUserPrompt` accepts | prompts | +| Queued prompt attachment of the main transcript that `isQueuedPrompt` accepts (a prompt sent while the agent works) | prompts | +| User record with an injected marker (command output, task notification, compact summary, command caveat) | none | +| `AskUserQuestion` result with an answer | questions_answered | +| `ExitPlanMode` result, not rejected | plans_approved | +| `ExitPlanMode` result, rejected | plans_rejected | +| Interrupt marker | interrupts | + +| File | Change | +|------|--------| +| `app/cli/internal/trace/claude/types.go` | Add to `jsonlRecord` the fields that the counter reads: `isMeta`, `toolUseResult`, `sourceToolUseID`, `permissionMode`, and the attachment fields. Add `tool_use_id` and `is_error` to the content block. | +| `app/cli/internal/trace/claude/interaction.go` | New: an accumulator with one method for each record. Use the rule of `isUserPrompt` and `isQueuedPrompt`, on the typed record. | +| `app/cli/internal/trace/claude/parse.go` | Feed the accumulator from the existing `scanJSONL` callbacks of `parseJSONL` and `parseSubagentUsage`, and set `HumanInteraction`. Make `scanJSONL` report the number of lines that it dropped. | +| `app/cli/internal/trace/claude/testdata/interaction-session.jsonl` | New fixture, cut from a real interactive session, with each touch point type and a mode change. | +| `app/cli/internal/trace/claude/interaction_test.go` | Table tests for each row of the rule table and for the mode order. Also: the subagent sum has no subagent prompts, an Esc at a plan dialog, and a record that cannot be read. | + +
+ +
+03 / 06 · Claude Code sessions record who started them (R-002, R-003) + +The parser reads the first user record of the main transcript. A record whose origin names another agent session gives `agent`. An origin that the parser does not know leaves out the value. Else, the run type field gives `automation` for a headless or SDK run, and `person` for an interactive terminal. The origin wins over the run type, so a headless run that an agent started gives `agent`. When the parser cannot tell, it leaves out the value. The first prompt counts in the prompt count only for `person` (R-002). + +| First user record | `session.started_by` | First prompt counted | +|-------------------|----------------------|----------------------| +| Origin is another agent session, headless or not | `agent` | no | +| Origin that the parser does not know | left out | no | +| Headless or SDK run, no agent origin | `automation` | no | +| Interactive terminal | `person` | yes | +| Cannot tell | left out | no | + +| File | Change | +|------|--------| +| `app/cli/internal/trace/claude/types.go` | Add the run type field and the origin field to `jsonlRecord`. | +| `app/cli/internal/trace/claude/interaction.go` | Find the starter at the first user record, in the same pass, and skip the first prompt when the starter is not a person. | +| `app/cli/internal/trace/claude/parse.go` | Set `Session.StartedBy`. | +| `app/cli/internal/trace/claude/testdata/interaction-headless.jsonl`, `app/cli/internal/trace/claude/testdata/interaction-agent-started.jsonl` | New fixtures from a real headless run and a real session that another agent session started. | +| `app/cli/internal/trace/claude/interaction_test.go` | Table tests for each row of the starter table. | + +
+ +
+04 / 06 · Claude Code records each permission request that it shows (R-005) + +`trace init` installs a `PermissionRequest` hook. The handler appends the tool name and the tool call ID of the request to a state file of the session. It writes nothing to stdout, so the hook makes no permission decision. The handler never fails the agent. It skips the question and plan tools (R-001). + +| File | Change | +|------|--------| +| `app/cli/internal/trace/claude/hooks.go` | Add the `PermissionRequest` event to `hookEvents`, with the `permission-request` command. | +| `app/cli/cmd/trace_hook.go` | Add the `claude permission-request` subcommand. | +| `app/cli/pkg/action/trace_agent_hook.go` | Add `HandleAgentPermissionRequest`: read the input, skip `AskUserQuestion` and `ExitPlanMode`, append the record. | +| `app/cli/internal/trace/state/permissions.go` | New: append and read the permission request records of a session. | +| `app/cli/internal/trace/claude/hooks_test.go`, `app/cli/cmd/trace_hook_test.go`, `app/cli/internal/trace/state/permissions_test.go` | Add tests: the hook installs and uninstalls, the command exists, records round-trip. | +| `app/cli/pkg/action/trace_agent_hook_test.go` | Add tests: the handler writes nothing to stdout, and it exits with success on input that it cannot read. | + +
+ +
+05 / 06 · Claude Code sessions record permission approvals and rejections (R-005, R-009, R-010) + +At push, the assembly reads the permission requests of the session and passes them to the parser in `ParseOpts`. For each request, the parser finds the tool result of the same call. A rejection marker gives a rejection. Any other result gives an approval, also a failed or interrupted run. No result gives neither. A request with no tool call ID cannot be matched safely, so it leaves out both counts and adds a warning (R-010). A session with no requests is normal, for example in an autonomous mode. So the session-start hook records in the session record whether the permission request hook was installed at that time. With the hook and no requests, both counts are zero. Without the hook, the parser leaves out both counts and adds a warning (R-009). + +| File | Change | +|------|--------| +| `app/cli/internal/trace/provider.go` | Add `PermissionRequests` and `PermissionHookInstalled` to `ParseOpts`. | +| `app/cli/internal/trace/claude/hooks.go` | Add a function that tells if the permission request hook is installed, next to `HooksInstalled`. | +| `app/cli/internal/trace/state/session.go`, `app/cli/pkg/action/trace_agent_hook.go` | At session start, call that function and store the result in the session record. | +| `app/cli/pkg/action/trace_hook_handler.go` | In `buildSessionEvidence`, read the requests and the stored result, and set both in `ParseOpts`. | +| `app/cli/internal/trace/claude/interaction.go` | Keep the tool results by call ID during the existing pass. After the pass, match each request to its result and count approvals and rejections. | +| `app/cli/internal/trace/claude/interaction_test.go` | Add cases for approvals: approved, and approved then failed. Add cases for rejections: with feedback text, and Esc at a dialog. Also add: unanswered, no tool call ID, no requests with the hook, and no hook. | + +
+ +
+06 / 06 · The README matrix shows which providers record the touch points (R-010) + +Add one row for each touch point type, one for the permission modes, and one for the starter to the Session collection table. Set the Claude Code cells to **Yes**. Set the other cells to **No**, with the "Why not Yes" line "Not implemented yet (spec issue-3829, milestone 2)". The e2e suite reports these rows as "no check" (TRC-026). That result is correct, because the rows have no check. + +| File | Change | +|------|--------| +| `app/cli/internal/trace/README.md` | Add the rows and the "Why not Yes" lines. The existing `TestTraceREADMEParses` test checks that the table still parses. | + +
+ +Verify: `go test ./pkg/attestation/crafter/materials/aicodingsession/... ./internal/schemavalidators/... ./app/cli/internal/trace/... ./app/cli/pkg/action/... ./app/cli/cmd/...` + +## Open Questions + +- [ ] **Does the Claude Code permission request hook carry the tool call ID?** Proposed: find out in step 04. Without an ID, step 05 leaves out the approval and rejection counts and adds a warning. It does not match by tool name, because two calls of one tool can overlap. + +## Skipped and risks + +| Skipped step | Reason | Follow-up | +|--------------|--------|-----------| +| End-to-end checks for the new README rows | The owner cut them from the spec. The rows report "no check". | none | +| A change to the existing user message count | The spec keeps its meaning (D-001). | none | +| A view in the platform UI | A Non-Goal of the spec. | platform ticket | +| Milestone 2 of the spec: Codex, Copilot, Cursor, OpenCode, and Pi | The owner limited this plan to Claude Code. | One plan for each provider, written when the team starts it | + +Not checked: some Claude Code field and marker names. No fixture in the repository holds them. They are `permissionMode`, the question and plan tool results, the rejection and interrupt markers, and the permission request hook payload. They also include the run type field and the origin of a message from another agent session. Steps 02, 03, and 04 start with real sessions for that reason: one interactive, one headless, and one that another agent session starts. + +| Risk | Mitigation | +|------|------------| +| A repository with old hooks has no permission request records. | The parser leaves out the approval and rejection counts and adds a warning (M1 step 05). | +| The permission request hook writes output that Claude Code reads as a decision. | The handler writes nothing to stdout. A test in step 04 checks it. | diff --git a/docs/specs/issue-3829-session-human-touch-points.md b/docs/specs/issue-3829-session-human-touch-points.md new file mode 100644 index 000000000..191b10ad9 --- /dev/null +++ b/docs/specs/issue-3829-session-human-touch-points.md @@ -0,0 +1,205 @@ +--- +owner: migmartri +ticket: https://github.com/chainloop-dev/chainloop/issues/3829 +prd: +--- + + +# Spec issue-3829: Record the human touch points of an AI coding session + +## Summary + +The AI coding session evidence will record how often a person stepped in during the session. Each type of human touch point gets its own count: prompts, answered questions, permission approvals and rejections, plan approvals and rejections, and interrupts. The evidence will also record who started the session, and the permission modes of the agent. The starter tells an interactive session that a person started from a headless or automated run. The permission mode tells which touch points the agent could ask for. The team needs this data to compare sessions and to improve autonomous agents. A later autonomy score can use it, but this spec does not define that score. + +## Problem + +- The session evidence does not record how often a person had to step in. We cannot tell an autonomous session from a session that a person guided step by step. +- The existing user message count does not measure human effort. For Claude Code it counts every record in the user role, which includes tool results and messages that the agent injects. +- The evidence does not record the permission mode of the agent. A session with no permission approvals can be an autonomous mode or a session with no risky actions. A reader cannot tell which. +- The evidence does not record who started the session. The start prompt of a scheduled headless run looks like a prompt that a person typed. +- The existing session mode (a coding session or a single `trace run`) is a different concept. It does not tell how the agent asked for permission. + +## Goals and Non-Goals + +- Goal: a reviewer can see how often a person stepped in during a session, for each type of touch point. +- Goal: when the provider records the starter and the counts, a reader can sort sessions into these groups: + - Started by another agent or an automated run, with no human touch point. + - Started by a person, with no human touch point after the first prompt. + - Started by a person, with human touch points after the first prompt. + - Started by another agent or an automated run, with human touch points later. +- In scope: the permission modes (R-008). They could be a separate spec, but the ticket names them, and a reader needs them to explain a count of zero. +- In scope: the counts of rejected permissions and plans (R-005, R-006) and of interrupts (R-007). The ticket names only approvals. A rejection and an interrupt are also times when a person stepped in, so they serve the same problem. +- In scope: who started the session (R-003). The ticket does not name it. But without it, the prompt count is wrong for a scheduled run, because its start prompt looks like a typed prompt. +- In scope: all providers (milestone 2). The team compares sessions across agents, and needs the same data for each agent. +- Non-goal: an autonomy score, or a change to the AI score. A later change can calculate one from this data. +- Non-goal: a common scale of permission modes across agents (for example ask, plan, autonomous). The score needs it, so the score work defines it. Follow-up: the autonomy score ticket. +- Non-goal: a view of the touch points in the platform UI. Follow-up: a platform ticket. +- Non-goal: the time that the agent waited for a person. +- Non-goal: a headless run that a person started by hand. The CLI cannot tell it from a run that a scheduler or a script started, so it records `automation` (R-003). +- Non-goal: whether the session completed its task. A session that a person stopped with no touch point looks the same as a session that completed. + +## Requirements + +### R-001: One count for each touch point, subagents included + +Each touch point MUST count in one count only. For example, the approval of a plan is not also a permission approval, and the answer to a question is not also a prompt. The touch points of a subagent MUST count in the totals of the session, because the user acted in that session. + +- Done when: the user presses Esc at a permission or plan dialog, and the evidence records one rejection and no interrupt. + +### R-002: User prompts + +The session evidence MUST count the messages that a person sent to the agent, typed or pasted. A slash command that a person typed counts. The first prompt counts only when a person started the session (R-003). Tool results, messages that the agent, a hook, or the CLI injects, and messages that another agent session sends MUST NOT count. + +- Done when: a session with three typed prompts, one of which holds a pasted link, records three prompts. The tool results and injected messages of the same session do not change the count. A scheduled headless run with one start prompt records zero prompts. + +### R-003: Who started the session + +The session evidence MUST record who started the session: a person, another agent session, or an automated run. An automated run is a run with no interactive user, for example a headless run that a scheduler or a script starts. Each headless run counts as an automated run. One exception: a run that another agent session started records `agent`, also when it is headless, because the sender of the first message wins. When the provider cannot tell, the evidence MUST leave out the value (R-009). + +- Done when: four sessions record their starters. An interactive session of a person records `person`. A session that another agent session started records `agent`, and so does a headless run that an agent started. A scheduled headless run records `automation`. + +### R-004: Answered questions + +The session evidence MUST count the questions that the agent asked through its question tool and that the user answered. A question with more than one part counts once. A question that the user closed with no answer does not count, because the ticket names answered questions. A message that the user then sends counts as a prompt. + +### R-005: Permission approvals and rejections + +The session evidence MUST count, in two counts, the permission requests that the user approved and the ones that the user rejected. Only requests that the agent showed to the user count. An action that the permission mode or an allow rule approved without a request MUST NOT count. An "always allow" answer counts as one approval. + +- Done when: a request that the user did not answer before the session ended counts in neither count. Text that the user typed with a rejection is part of the rejection, not a prompt. + +### R-006: Plan approvals and rejections + +The session evidence MUST count, in two counts, the plans that the user approved and the ones that the user rejected. + +- Done when: text that the user typed with a plan rejection is part of the rejection, not a prompt. + +### R-007: Interrupts + +The session evidence MUST count the times that the user interrupted the agent while it worked. An interrupt MUST NOT also count as a prompt. + +- Done when: a message that the user sends after an interrupt counts as a prompt. + +### R-008: Permission modes + +The session evidence MUST record each permission mode that the agent used in the session, once each, in the order of first use. The start mode alone hides a later change to an autonomous mode, and the order shows both. Each value MUST be the name that the agent itself uses for the permission mode. + +- Done when: a Claude Code session starts in the default mode and then changes to the auto mode. The evidence records both permission modes, default first. + +### R-009: Not recorded is different from zero + +When a provider can detect a touch point type, the evidence MUST hold its count, also when the count is zero. When a provider cannot detect a type, the evidence MUST leave out that count. The same rule applies to the permission modes. + +### R-010: Provider support, compatibility, and failure safety + +Each provider MUST record the touch point types and the permission modes that its transcript or hook events show. Claude Code MUST record all of them. Sometimes a provider cannot tell an approval by the user from an approval by a rule. Then it MUST leave out the approval count, and MUST NOT guess. The new data MUST be additive, and the schema version of the evidence MUST stay the same. The existing conversation counts MUST keep their meaning. A failure to count MUST give a warning, MUST leave out the affected counts, and MUST NOT stop evidence creation or block the Git push. + +- Done when: old evidence and new evidence both pass schema validation. The compatibility matrix in the Trace README gets new rows. Each touch point type, the permission modes, and the starter get one row each. Each row has a cell for each provider. Each cell that is not **Yes** has a "Why not Yes" line. The matrix changes in the same PR as the provider code. Its cells are the expected result for each provider, and the PR of the milestone for that provider sets them. + +## Constraints + +- The repository is public. The evidence format is visible to all users, and readers on the platform store evidence from older CLIs. +- The transcript and hook formats of the agents are not stable interfaces. +- The hooks meet the same baseline as the existing Trace hooks. Counting adds no step that waits for the network or reads the full transcript in a hook. + +## Proposal + +The user works with the agent as before. Nothing changes in the agent. When the user pushes, the session evidence holds a new human interaction section next to the existing conversation counts. + +| Field | Meaning | +|-------|---------| +| `permission_modes` | The permission modes that the agent used, in the order of first use (R-008) | +| `touch_points.prompts` | Messages that a person sent (R-002) | +| `touch_points.questions_answered` | Questions of the agent that the user answered (R-004) | +| `touch_points.permissions_approved` | Permission requests that the user approved (R-005) | +| `touch_points.permissions_rejected` | Permission requests that the user rejected (R-005) | +| `touch_points.plans_approved` | Plans that the user approved (R-006) | +| `touch_points.plans_rejected` | Plans that the user rejected (R-006) | +| `touch_points.interrupts` | Interrupts of the agent by the user (R-007) | + +The existing session object also gets a `started_by` field, next to the session mode, with the values `person`, `agent`, or `automation` (R-003). It describes the session, not a human action, so it is not part of the human interaction section. + +The CLI makes this section at push time, in the same step that parses the transcript for the other session data. The provider of each agent counts the touch points, because each agent records them differently. A provider that cannot detect a type leaves out its field (R-009). + +**Claude Code.** The transcript records the permission mode on each user record, and the hook events carry it too. The transcript shows the prompts, the answers to the question tool, the result of each plan tool call, and the interrupt markers. A rejected permission request gives a tool result that says the user rejected the tool use. The transcript does not show whether the agent asked before an approved action, because an allow rule gives the same record. The Trace hooks will also listen to the event that Claude Code sends when it shows a permission request. A request with a tool result that is not a rejection is an approval, also when the tool then failed or the user interrupted it. Requests for the plan tool and the question tool are not permission requests (R-001). For the starter, the transcript records how the agent ran: an interactive terminal, or a headless or SDK run. A message from another agent session carries an origin that is not a person. The first user record of the session gives the starter (R-003). + +**Other providers.** Each provider records the types that its transcript or hook events show, for example the approval policy of Codex. It leaves out the other types (R-009). The compatibility matrix in the Trace README records the result for each provider and type (R-010). + +```mermaid +flowchart LR + U[User] -->|prompts, answers, approvals, rejections, interrupts| A[AI coding agent] + A -->|transcript| T[Transcript copy] + A -->|hook events, including permission requests| H[Trace hooks] + H --> S[Trace state] + T --> P[Provider parser at push time] + S --> P + P -->|permission modes and touch point counts| E[Session evidence: human interaction section] + E --> V[Schema validation in the CLI] +``` + +## Decision Record + +### D-001: A new section, and the conversation counts stay the same + +The touch points and the permission modes go into a new human interaction section. The existing user message count keeps its meaning. + +- Why: a change to the meaning of the user message count makes old and new evidence disagree, and platform readers store both. A separate section keeps the human data apart from the message statistics. +- Alternatives considered: + - Change the user message count to count prompts only: stored evidence then has two meanings for one field. + - Add the counts to the conversation section: the new prompt count sits next to a user message count with a different meaning. +- Source: drafting + +### D-002: Native permission mode names, with no common scale + +The evidence stores the permission mode name of the agent, for example `acceptEdits` or `bypassPermissions` for Claude Code. + +- Why: a mapping to a common scale can be wrong, and it loses detail that the score work can need. The score work can map the native names later, also for evidence that is already stored. +- Alternatives considered: + - A common scale only: needs a mapping for each agent now, and signed evidence keeps a wrong mapping. + - Both values: more scope, and the common scale is not defined yet. +- Source: drafting + +### D-003: Permission requests come from a hook event, not only from the transcript + +For Claude Code, the CLI finds a permission request from the hook event that the agent sends when it shows the request. It does not guess from the transcript. Follows TRC-009: the CLI makes this data from the transcript and the hook events, and the agent does not report it. + +- Why: an approval by the user and an approval by an allow rule give the same transcript records. Only the request event tells them apart. +- Alternatives considered: + - Count each completed tool call that the permission mode does not approve: needs all the allow rules that applied at that time. A wrong guess counts actions that nobody approved. + - Ask the agent to report its touch points: the agent does not see the permission dialog. Also, the count must not depend on the model. +- Source: drafting + +## Open Questions + +None. + +## Milestones + +1. **Evidence section and Claude Code.** The evidence and its schema get the new section. Claude Code records all the counts and the permission modes. The README matrix gets the new rows. +2. **One milestone for each other provider:** Codex, Copilot, Cursor, OpenCode, and Pi. Each one records the types that its agent shows, and updates its cells in the README matrix. These milestones can ship in any order after milestone 1. + +## Risks + +| Risk | Mitigation | +|------|------------| +| A new Claude Code version changes a transcript marker or the permission request event. The provider then counts zero, and it cannot tell that the count is wrong. | We accept this risk. The counts are data for a later score, not a gate, and a wrong count does not block a push. | +| Readers compare permission modes across agents as if the names had one meaning. | The README and the evidence schema say that the names are native to each agent (D-002). |