Repository navigation
Conversation
Design spec for recording, in the AI coding session evidence, how often a person stepped in during the session, and the permission modes of the agent. Refs chainloop-dev#3829 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
Bring all providers into the scope of the spec, with one milestone for each, and tighten the counting rules. Add the implementation plan for milestone 1: the evidence section and Claude Code. Refs chainloop-dev#3829 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
PR validation —
|
| Status | Policy | Material | Messages |
|---|---|---|---|
pr-min-approvals |
pr-info |
PR/MR #3831 has 0 approving reviews, 1 required. | |
| ✅ Passed | pr-description-required |
pr-info |
- |
| ✅ Passed | pr-user-story-linked |
pr-info |
- |
AI Session Checks — 🟡 82% · ✅ 0 failing
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟡 82% | 1 | ✅ 0 | 100% AI / 0% Human | 2 | +401 / -0 | 21h49m20s |
🟡 82% — 100% AI — ✅ All policies passing
-
Nov 18, 2026 11:32 UTC · 21h49m20s · $97.83 · 1.7k in / 727.0k out · claude-code 2.1.327 (claude-opus-5-5)
Change Summary
-
Adds a design spec and a single-provider milestone plan for recording session human touch points; no runtime behavior changes.
AI Session Overall Score
-
🟡 82% — Documentation stays focused, but compatibility assumptions and scope clarification required user intervention; runtime criteria abstained.
AI Session Analysis Breakdown
-
🟢 96% · scope-discipline
-
🟢 Only the design spec and milestone plan changed; no scoring, runtime, or unrelated files were added. · High Impact
🟢 90% · context-and-planning
-
🟢 Before drafting, the agent inspected conventions, existing evidence semantics, and transcript records, then asked explicit design questions. · High Impact
🟡 78% · user-trust-signal
-
🟠 After interrupting competing scope options, the user restated that the spec stays broad while the plan covers one provider. · Medium Severity
💡 When choices blur spec and plan scope, ask for a two-line restatement before further edits.
🟡 72% · alignment
-
🟢 The delivered proposal specifies separate prompt, answered-question, permission-approval, and plan-approval counts. · High Impact
🟠 The agent asserted old servers reject new evidence, then admitted assuming this when the user challenged it. · Medium Severity
💡 Trace validation callers before presenting release-order assumptions as established facts.
abstained · solution-quality
-
🟡 solution-quality not assessed — reviewer should confirm coverage · Low Severity
abstained · verification
-
🟡 verification not assessed — reviewer should confirm coverage · Low Severity
-
File Attribution
████████████████████100% AI / 0% HumanStatus Attribution File Lines created ai docs/specs/issue-3829-session-human-touch-points.md+205 / -0 created ai docs/plans/issue-3829-session-human-touch-points.md+196 / -0
Policies (4)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-bb711c- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-bb711c- ✅ Passed ai-config-no-secretsai-coding-session-bb711c- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-bb711c- -
Security Checks — ✅ 5 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
Scans not applied (3)
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
iac-scan |
no IaC files changed |
Powered by Chainloop and Chainloop Trace
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
Add session.started_by (person, agent, or automation) so a reader can tell sessions that another agent or an automated run started from sessions that a person started. The first prompt counts as a human prompt only when a person started the session. Add the matching plan step for Claude Code. Refs chainloop-dev#3829 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
|
@jiparis could you take a look? This spec adds human touch point counts, the permission modes, and |
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
Count an approval also when the approved tool fails, clarify what the starter tells apart, and state that only the CLI validates the session evidence. In the plan, count queued prompts, store at session start whether the permission hook is installed, leave out approval counts when a request has no tool call ID, omit an unknown starter, and give 'agent' only for a known agent origin. Refs chainloop-dev#3829 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
The Claude Code counter is an accumulator fed from the existing scanJSONL callbacks, so the parser reads each transcript once and adds no second loop. The typed record gets the fields that the counter reads, and scanJSONL reports dropped lines so the counts are left out with a warning. Refs chainloop-dev#3829 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/plans/issue-3829-session-human-touch-points.md">
<violation number="1" location="docs/plans/issue-3829-session-human-touch-points.md:88">
P2: A dropped user record can hide a permission mode or the actual first record, but this fallback only omits touch-point counts. Track that uncertainty and omit `permission_modes` and `started_by` (and do not classify the next prompt as the first prompt) when a dropped line makes them unknowable.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
View guided diff | Re-trigger cubic
| <details> | ||
| <summary><strong>02 / 06 · Claude Code sessions record prompts, questions, plans, interrupts, and permission modes (R-001, R-002, R-004, R-005, R-006, R-008)</strong></summary> | ||
|
|
||
| The parser already reads each transcript once: `scanJSONL` decodes each record into `jsonlRecord` and calls back `parseJSONL` for the main transcript and `parseSubagentUsage` for each subagent transcript. The counter adds no second read and no second loop. It is an accumulator that these two callbacks feed with each record, and `jsonlRecord` gets the fields that it reads. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-008). The permission modes come from the `permissionMode` field of the user records, in the order of first use. `scanJSONL` drops a line that it cannot decode before the callback runs. So it will also report how many lines it dropped. When it dropped a line, the parser leaves out the touch point counts and adds a warning (R-009). |
There was a problem hiding this comment.
P2: A dropped user record can hide a permission mode or the actual first record, but this fallback only omits touch-point counts. Track that uncertainty and omit permission_modes and started_by (and do not classify the next prompt as the first prompt) when a dropped line makes them unknowable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/plans/issue-3829-session-human-touch-points.md, line 88:
<comment>A dropped user record can hide a permission mode or the actual first record, but this fallback only omits touch-point counts. Track that uncertainty and omit `permission_modes` and `started_by` (and do not classify the next prompt as the first prompt) when a dropped line makes them unknowable.</comment>
<file context>
@@ -85,7 +85,7 @@ Add the section to the evidence data with the fields of the spec Proposal table.
<summary><strong>02 / 06 · Claude Code sessions record prompts, questions, plans, interrupts, and permission modes (R-001, R-002, R-004, R-005, R-006, R-008)</strong></summary>
-The parser reads each record of the main transcript and of each subagent transcript. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-008). The permission modes come from the `permissionMode` field of the user records, in the order of first use. A record that the parser cannot read gives a warning, and the parser leaves out the counts that it affects (R-009).
+The parser already reads each transcript once: `scanJSONL` decodes each record into `jsonlRecord` and calls back `parseJSONL` for the main transcript and `parseSubagentUsage` for each subagent transcript. The counter adds no second read and no second loop. It is an accumulator that these two callbacks feed with each record, and `jsonlRecord` gets the fields that it reads. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-008). The permission modes come from the `permissionMode` field of the user records, in the order of first use. `scanJSONL` drops a line that it cannot decode before the callback runs. So it will also report how many lines it dropped. When it dropped a line, the parser leaves out the touch point counts and adds a warning (R-009).
| Rule input | Count |
</file context>
| - Ask the agent to report its touch points: the agent does not see the permission dialog. Also, the count must not depend on the model. | ||
| - Source: drafting | ||
|
|
||
| ### D-006: Drops D-003 and D-005 |
There was a problem hiding this comment.
Please remove this and renumber the decisions
There was a problem hiding this comment.
Done in 71c288e. The entry is removed, and the decisions are now D-001 to D-003 (old D-004 is now D-003).
|
|
||
| ## Requirements | ||
|
|
||
| ### R-008: One count for each touch point, subagents included |
There was a problem hiding this comment.
Why is 8 before 1? please renumber them
There was a problem hiding this comment.
Done in 71c288e. The requirements now run R-001 to R-010 in document order. Earlier review threads on this PR cite the old IDs, so here is the map (old → new): R-008 → R-001, R-001 → R-002, R-010 → R-003, R-002 → R-004, R-003 → R-005, R-004 → R-006, R-005 → R-007, R-006 → R-008, R-007 → R-009, R-009 → R-010. Decisions: D-004 → D-003.
| flowchart LR | ||
| A[Agent] -->|transcript| C[Transcript copy] | ||
| A -->|permission request hook, Claude Code only| S[Trace state] | ||
| C --> P[Provider parser at push] |
There was a problem hiding this comment.
Since the info can be derived from the transcript, did you take into consideration using post processing server side for this "statistical" data? We may want other data in the future, and this process would require changing the format of the evidence every time.
There was a problem hiding this comment.
Good question. We checked what the server could derive from today's upload, for each agent. The results and a possible hybrid are in #3831 (comment). We will decide after that discussion.
What I'm wondering is whether this could be done server side instead. Mostly if we want to add more metrics in the future, it would require schema changes. |
Number the requirements R-001 to R-010 in document order and the decisions D-001 to D-003, and remove the entry that recorded the dropped decision IDs, as asked in review. The plan references follow the new IDs. Refs chainloop-dev#3829 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
|
@jiparis about the derivation on the server: we checked what the server could derive from the evidence that the CLI already uploads ( Can the server derive each item from today's upload?
What this shows:
A possible hybrid: this spec changes to "the evidence carries the raw facts". The CLI records the permission request events from the hooks, and the uploads stop dropping native records. The platform then derives the counts, the modes, and the starter, and a new metric needs no schema change. The cost is a transcript parser for each agent on the server, and no statistics in local mode. We have not decided yet. What do you think? |
|
it's clear then, if it cannot be done server side, it cannot be done server side. |
|
@jiparis thanks. We are going to re-evaluate this and come back with a decision on where the counting lives. |
Summary
This PR adds a design spec and its implementation plan for #3829.
The AI coding session evidence will record how often a person stepped in during the session. Each type of human touch point gets its own count: prompts, answered questions, permission approvals and rejections, plan approvals and rejections, and interrupts. The evidence will also record who started the session (
session.started_by: a person, another agent, or an automated run), and the permission modes of the agent. The team needs this data to compare sessions and to improve autonomous agents. With it, a reader can tell apart:sessions that another agent or an automated run started, with no human touch point,
sessions that a person started, with no human touch point after the first prompt,
sessions that a person started, with human touch points after the first prompt. A later autonomy score can use it, but this spec does not define that score.
Spec:
docs/specs/issue-3829-session-human-touch-points.md. All providers are in scope, with one milestone for each provider.Plan:
docs/plans/issue-3829-session-human-touch-points.md. It covers milestone 1 only: the evidence section and Claude Code.Open questions for reviewers
We would like two reviewers to approve this spec before it merges.
AI assistance: Claude Code helped draft the spec and the plan.
Refs #3829