Skip to content

D237 — a failed module check says its tool's last lines and keeps its whole output - #433

Merged
napkinstack-agent[bot] merged 1 commit into
mainfrom
d237-failed-output-kept
Nov 18, 2026
Merged

napkinstack-agent[bot] merged 1 commit into
mainfrom
d237-failed-output-kept

Conversation

@napkinstack-agent

Copy link
Copy Markdown
Contributor

Closes #428.

Why

On #428's project, a test failed under nstack attest, and the only output was FAIL [test] module 'shell': `cargo test --locked …` exited with 101. Run again by hand, the test passed: it was flaky, and its name was lost, so there was nothing to quarantine or to file. To get the name the next time, the team put a cargo shim on the attest's PATH.

What

  • Under such a FAIL line, attest shows the last lines (8) the module's own command wrote before it exited. A test runner's summary (cargo, pytest, Go) names the failed test there.
    • The window opens at the engine's -> <module>: <command>.
    • It freezes during the engine's evidence audit (Evidence of …), whose lines are the engine's, not the tool's.
    • It empties once said, so a nested run's lines are shown once.
  • The check's whole output (its last 8 MiB) is kept in the clone's git folder: .git/nstack/attest/<commit>-<check>.log, or the worktree's own git folder in a worktree. Its path is printed under the FAIL line.
    • It is never in the tree, where it would leave changes the next attest refuses. It is never pushed.
    • A disk that cannot take it says so and never costs the record.
  • A check that passes keeps nothing new. The record is unchanged: 64 KiB of each step, the whole hashed.
  • Older bug fixed on the way: output that is not UTF-8 no longer crashes the attest; it is read with replacement.
  • 07-governance.md §7 says it. Register row D237.

Tests

  • test_a_failed_module_check_says_its_tool_s_last_lines_and_keeps_its_output (red on main): a cargo-like failure with 400 lines of noise.
    • The failed test's name and the summary are shown; the noise is not.
    • The log holds the whole output.
    • The tree stays clean.
  • test_a_passed_check_leaves_no_log.
  • test_each_module_s_failure_says_its_own_tool_lines_and_never_the_engine_s:
    • a module that passed before says nothing under another module's failure;
    • an e2e's evidence audit is not taken for its tool's lines;
    • a nested run's lines are said once.
  • test_a_step_that_prints_bytes_not_utf_8_is_recorded (red on main).
  • Oracle: platform tests OK. Fitness compliant.
  • Independent verifier: nothing newly published, since the tool lines are already in the record and the log is local. Its findings are fixed:
    • the row's place;
    • the e2e audit lines displaced the tool's;
    • --path-format=absolute on git older than 2.31 wrote the log into the tree;
    • a nested run's lines were said twice;
    • a log that could not be written crashed the attest;
    • output that is not UTF-8 crashed the attest.
  • Not taken from the verifier:
    • a multi-line manifest command, whose FAIL line spans lines, still gets no tool lines. It is no worse than before.
    • ::error annotations are not counted as the tool's lines.

🤖 Generated with Claude Code

… whole output

Closes #428.

Agent-Session: nstack-os-d237

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@napkinstack-agent
napkinstack-agent Bot requested a review from a team as a code owner November 18, 2026 10:29
@napkinstack-agent
napkinstack-agent Bot merged commit 08575a9 into main Nov 18, 2026
3 checks passed
@napkinstack-agent
napkinstack-agent Bot deleted the d237-failed-output-kept branch November 18, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A failed module check keeps only its exit code: a flaky test that passes again is never named

1 participant