Skip to content

spec: issue-3829 Record the human touch points of an AI coding session - #3831

Draft
migmartri wants to merge 6 commits into
chainloop-dev:mainfrom
migmartri:issue-3829-spec-session-human-touch-points
Draft

migmartri wants to merge 6 commits into
chainloop-dev:mainfrom
migmartri:issue-3829-spec-session-human-touch-points

Conversation

@migmartri

@migmartri migmartri commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR adds a design spec and its implementation plan for #3829.

The AI coding session evidence will record how often a person stepped in during the session. Each type of human touch point gets its own count: prompts, answered questions, permission approvals and rejections, plan approvals and rejections, and interrupts. The evidence will also record who started the session (session.started_by: a person, another agent, or an automated run), and the permission modes of the agent. The team needs this data to compare sessions and to improve autonomous agents. With it, a reader can tell apart:

  • sessions that another agent or an automated run started, with no human touch point,

  • sessions that a person started, with no human touch point after the first prompt,

  • sessions that a person started, with human touch points after the first prompt. A later autonomy score can use it, but this spec does not define that score.

  • Spec: docs/specs/issue-3829-session-human-touch-points.md. All providers are in scope, with one milestone for each provider.

  • Plan: docs/plans/issue-3829-session-human-touch-points.md. It covers milestone 1 only: the evidence section and Claude Code.

Open questions for reviewers

  • Spec: none.
  • Plan: does the Claude Code permission request hook carry the tool call ID? Proposed: find out in step 03. Without an ID, match each request to the next result of the same tool.

We would like two reviewers to approve this spec before it merges.

AI assistance: Claude Code helped draft the spec and the plan.

Refs #3829

Design spec for recording, in the AI coding session evidence, how often a
person stepped in during the session, and the permission modes of the agent.

Refs chainloop-dev#3829

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
Bring all providers into the scope of the spec, with one milestone for
each, and tighten the counting rules. Add the implementation plan for
milestone 1: the evidence section and Claude Code.

Refs chainloop-dev#3829

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
@migmartri migmartri added the spec Design spec label Nov 18, 2026
@chainloop-platform

chainloop-platform Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ⚠️ 1 failing

Status Policy Material Messages
⚠️ Failed pr-min-approvals pr-info PR/MR #3831 has 0 approving reviews, 1 required.
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — 🟡 82% · ✅ 0 failing

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟡 82% 1 ✅ 0 100% AI / 0% Human 2 +401 / -0 21h49m20s

🟡 82% — 100% AI — ✅ All policies passing

Nov 18, 2026 11:32 UTC · 21h49m20s · $97.83 · 1.7k in / 727.0k out · claude-code 2.1.327 (claude-opus-5-5)

View session details ↗

Change Summary

Adds a design spec and a single-provider milestone plan for recording session human touch points; no runtime behavior changes.

AI Session Overall Score

🟡 82% — Documentation stays focused, but compatibility assumptions and scope clarification required user intervention; runtime criteria abstained.

AI Session Analysis Breakdown

🟢 96% · scope-discipline

🟢 Only the design spec and milestone plan changed; no scoring, runtime, or unrelated files were added. · High Impact

🟢 90% · context-and-planning

🟢 Before drafting, the agent inspected conventions, existing evidence semantics, and transcript records, then asked explicit design questions. · High Impact

🟡 78% · user-trust-signal

🟠 After interrupting competing scope options, the user restated that the spec stays broad while the plan covers one provider. · Medium Severity

💡 When choices blur spec and plan scope, ask for a two-line restatement before further edits.

🟡 72% · alignment

🟢 The delivered proposal specifies separate prompt, answered-question, permission-approval, and plan-approval counts. · High Impact

🟠 The agent asserted old servers reject new evidence, then admitted assuming this when the user challenged it. · Medium Severity

💡 Trace validation callers before presenting release-order assumptions as established facts.

abstained · solution-quality

🟡 solution-quality not assessed — reviewer should confirm coverage · Low Severity

abstained · verification

🟡 verification not assessed — reviewer should confirm coverage · Low Severity


File Attribution

████████████████████ 100% AI / 0% Human

Status Attribution File Lines
created ai docs/specs/issue-3829-session-human-touch-points.md +205 / -0
created ai docs/plans/issue-3829-session-human-touch-points.md +196 / -0

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-bb711c -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-bb711c -
✅ Passed ai-config-no-secrets ai-coding-session-bb711c -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-bb711c -

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Add session.started_by (person, agent, or automation) so a reader can
tell sessions that another agent or an automated run started from
sessions that a person started. The first prompt counts as a human
prompt only when a person started the session. Add the matching plan
step for Claude Code.

Refs chainloop-dev#3829

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
@migmartri

Copy link
Copy Markdown
Member Author

@jiparis could you take a look? This spec adds human touch point counts, the permission modes, and session.started_by to the AI coding session evidence. The plan covers the Claude Code milestone. You own the Chainloop Trace area, so your review of the provider and hook parts would help.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Count an approval also when the approved tool fails, clarify what the
starter tells apart, and state that only the CLI validates the session
evidence. In the plan, count queued prompts, store at session start
whether the permission hook is installed, leave out approval counts
when a request has no tool call ID, omit an unknown starter, and give
'agent' only for a known agent origin.

Refs chainloop-dev#3829

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
The Claude Code counter is an accumulator fed from the existing
scanJSONL callbacks, so the parser reads each transcript once and adds
no second loop. The typed record gets the fields that the counter
reads, and scanJSONL reports dropped lines so the counts are left out
with a warning.

Refs chainloop-dev#3829

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/plans/issue-3829-session-human-touch-points.md">

<violation number="1" location="docs/plans/issue-3829-session-human-touch-points.md:88">
P2: A dropped user record can hide a permission mode or the actual first record, but this fallback only omits touch-point counts. Track that uncertainty and omit `permission_modes` and `started_by` (and do not classify the next prompt as the first prompt) when a dropped line makes them unknowable.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

View guided diff | Re-trigger cubic

<details>
<summary><strong>02 / 06 · Claude Code sessions record prompts, questions, plans, interrupts, and permission modes (R-001, R-002, R-004, R-005, R-006, R-008)</strong></summary>

The parser already reads each transcript once: `scanJSONL` decodes each record into `jsonlRecord` and calls back `parseJSONL` for the main transcript and `parseSubagentUsage` for each subagent transcript. The counter adds no second read and no second loop. It is an accumulator that these two callbacks feed with each record, and `jsonlRecord` gets the fields that it reads. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-008). The permission modes come from the `permissionMode` field of the user records, in the order of first use. `scanJSONL` drops a line that it cannot decode before the callback runs. So it will also report how many lines it dropped. When it dropped a line, the parser leaves out the touch point counts and adds a warning (R-009).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A dropped user record can hide a permission mode or the actual first record, but this fallback only omits touch-point counts. Track that uncertainty and omit permission_modes and started_by (and do not classify the next prompt as the first prompt) when a dropped line makes them unknowable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/plans/issue-3829-session-human-touch-points.md, line 88:

<comment>A dropped user record can hide a permission mode or the actual first record, but this fallback only omits touch-point counts. Track that uncertainty and omit `permission_modes` and `started_by` (and do not classify the next prompt as the first prompt) when a dropped line makes them unknowable.</comment>

<file context>
@@ -85,7 +85,7 @@ Add the section to the evidence data with the fields of the spec Proposal table.
 <summary><strong>02 / 06 · Claude Code sessions record prompts, questions, plans, interrupts, and permission modes (R-001, R-002, R-004, R-005, R-006, R-008)</strong></summary>
 
-The parser reads each record of the main transcript and of each subagent transcript. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-008). The permission modes come from the `permissionMode` field of the user records, in the order of first use. A record that the parser cannot read gives a warning, and the parser leaves out the counts that it affects (R-009).
+The parser already reads each transcript once: `scanJSONL` decodes each record into `jsonlRecord` and calls back `parseJSONL` for the main transcript and `parseSubagentUsage` for each subagent transcript. The counter adds no second read and no second loop. It is an accumulator that these two callbacks feed with each record, and `jsonlRecord` gets the fields that it reads. Prompts come from the main transcript only, because the first message of a subagent comes from the parent agent. A prompt is a record that the existing `isUserPrompt` function accepts, with no interrupt marker and no injected marker. A question is an `AskUserQuestion` tool result with an answer. A plan is an `ExitPlanMode` tool result: approved when it is not a rejection. An interrupt is the interrupt marker text. The interrupt marker that follows a rejection at a plan or permission dialog is not an interrupt (R-008). The permission modes come from the `permissionMode` field of the user records, in the order of first use. `scanJSONL` drops a line that it cannot decode before the callback runs. So it will also report how many lines it dropped. When it dropped a line, the parser leaves out the touch point counts and adds a warning (R-009).
 
 | Rule input | Count |
</file context>

- Ask the agent to report its touch points: the agent does not see the permission dialog. Also, the count must not depend on the model.
- Source: drafting

### D-006: Drops D-003 and D-005

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this and renumber the decisions

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 71c288e. The entry is removed, and the decisions are now D-001 to D-003 (old D-004 is now D-003).


## Requirements

### R-008: One count for each touch point, subagents included

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is 8 before 1? please renumber them

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 71c288e. The requirements now run R-001 to R-010 in document order. Earlier review threads on this PR cite the old IDs, so here is the map (old → new): R-008 → R-001, R-001 → R-002, R-010 → R-003, R-002 → R-004, R-003 → R-005, R-004 → R-006, R-005 → R-007, R-006 → R-008, R-007 → R-009, R-009 → R-010. Decisions: D-004 → D-003.

flowchart LR
A[Agent] -->|transcript| C[Transcript copy]
A -->|permission request hook, Claude Code only| S[Trace state]
C --> P[Provider parser at push]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the info can be derived from the transcript, did you take into consideration using post processing server side for this "statistical" data? We may want other data in the future, and this process would require changing the format of the evidence every time.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. We checked what the server could derive from today's upload, for each agent. The results and a possible hybrid are in #3831 (comment). We will decide after that discussion.

@jiparis

jiparis commented Nov 18, 2026

Copy link
Copy Markdown
Member

@jiparis could you take a look? This spec adds human touch point counts, the permission modes, and session.started_by to the AI coding session evidence. The plan covers the Claude Code milestone. You own the Chainloop Trace area, so your review of the provider and hook parts would help.

What I'm wondering is whether this could be done server side instead. Mostly if we want to add more metrics in the future, it would require schema changes.

Number the requirements R-001 to R-010 in document order and the
decisions D-001 to D-003, and remove the entry that recorded the
dropped decision IDs, as asked in review. The plan references follow
the new IDs.

Refs chainloop-dev#3829

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: bb711ca3-8e66-4519-94d3-c1a967a30914
@migmartri

Copy link
Copy Markdown
Member Author

@jiparis about the derivation on the server: we checked what the server could derive from the evidence that the CLI already uploads (raw_session). We read the provider code, and we counted signals in real local sessions of Claude Code, Codex, Copilot, OpenCode 2, and Pi. Cursor and OpenCode 1.x are from the code only.

Can the server derive each item from today's upload?

Item Claude Codex Copilot Cursor OpenCode Pi
Prompts yes yes yes partly partly (synthetic messages mixed in) yes
Started by yes yes (session_meta) partly no no (dropped by the rebuild) no (header dropped)
Answered questions yes yes yes no no (answer dropped) n/a
Plan approve/reject unconfirmed partly (plan mode only) partly no no (dropped) n/a
Permission rejections yes no samples partly no no n/a
Permission approvals by the user no, needs a hook no, needs a hook yes (decisionSource) no no n/a
Interrupts yes yes (turn_aborted) yes no no (dropped) yes
Permission modes yes yes (approval_policy) yes no no (dropped) n/a

What this shows:

  1. Claude Code, Codex, and Copilot keep their native records in the upload. So the server can derive almost everything for them.
  2. Several "no" cells come from our own upload code, not from the agent. The OpenCode rebuild drops interrupts, agent switches, question answers, and the synthetic flag. Pi drops the session header. If these uploads keep the native records, as Codex does with source_event, those cells become derivable.
  3. Permission approvals by the user are not in the Claude Code or the Codex transcript. Only a hook can record them.
  4. Cursor is weak on both sides. Its transcript holds only the role and the text.
  5. The CLI and the server see the same transcript. The only extra input of the CLI is the hook events. So the choice is where the per-agent parsing lives.

A possible hybrid: this spec changes to "the evidence carries the raw facts". The CLI records the permission request events from the hooks, and the uploads stop dropping native records. The platform then derives the counts, the modes, and the starter, and a new metric needs no schema change. The cost is a transcript parser for each agent on the server, and no statistics in local mode.

We have not decided yet. What do you think?

jiparis commented Nov 19, 2026

Copy link
Copy Markdown
Member

it's clear then, if it cannot be done server side, it cannot be done server side.

@migmartri
migmartri marked this pull request as draft November 19, 2026 10:37
@migmartri

migmartri commented Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

@jiparis thanks. We are going to re-evaluate this and come back with a decision on where the counting lives.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spec Design spec

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants