Close out the cutover: spec §6 step 5 done, and re-decide D5 (lockstep pin, Watcher-generated goldens) now that Watcher doesn't extract #47

Activity

1. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Hand-off from the burn-down epic #24, 2026-10-10. This is next. #45 shipped in #48.

   Why now. watcher#350 (closed 2026-10-10T01:52Z) deleted Watcher's extraction. Two rules every implementing thread reads first, AGENTS.md lines 21 and 55, now guard nothing, and the goldens generator no longer runs on Watcher's main. #1 closes on this item.

   Operator decisions (2026-10-10, at the check-in):

   1. D5: Processor owns the co-core pin. Extractor bumps are Processor's call. Coordinate with Watcher only when a bump changes the contract: the content.process / content.derived models, canonical_text semantics, or resolve_dispatch_essence. Give Watcher notice whenever processor_version moves, because Watcher re-baselines on it (Option A: a processor change alone re-baselines silently and audits check.rebaselined). Watcher still pins co-core==0.19.7 for contracts and the bus. That pin no longer has to equal Processor's.
   2. Goldens: from co-core directly. Freeze today's tests/fixtures/parity/goldens.json as the v1 record. At a bump, regenerate from co-core's pure extract API directly, never through Processor's wrapper (processors/extract.py), and list every changed digest in the bump note as a deliberate re-baseline. The real corpus (tests/fixtures/parity/real/, Real-input parity corpus from Watcher's export; record watcher#325's answers in the spec #16) stays Watcher's export verbatim.

   Measured, 2026-10-10:

   • scripts/gen_parity_goldens.py imports src.core.registry.ServiceRegistry, src.core.media_type.extraction_overrides_for_essence and src.workers.pipeline._extract_and_fingerprint. watcher#350 deleted all three. It runs only inside a Watcher checkout from before 207e9be.
   • goldens.json metadata: co_core: 0.19.7, generated_by: … (Watcher's _extract_and_fingerprint), watcher_commit: a5d6f34…, watcher_processor_version: 0.19.7+1.

   Touchpoints:

   • Spec:
     • line 87 (D5's row): restate it as decision 1, keeping the history (amended 2026-10-10, watcher#350);
     • §5 (versioning and the pin);
     • §6 cutover order step 5 (around line 445): mark it done, with links to watcher#326 (the gate verdict 2026-10-06, the soak verdict 2026-10-09) and watcher#350;
     • §7 (testing, parity): the new golden source.
   • AGENTS.md: line 21 ("in lockstep with Watcher") and line 55 ("Parity goldens come from Watcher's own code"). Keep the warning that a failing parity test is never fixed by regenerating.
   • DEPLOYMENT.md:459: the bump procedure, step 2 ("Regenerate the goldens from Watcher on the new version").
   • scripts/gen_parity_goldens.py: replace it with a co-core-direct generator (or retire it and add a new one; say which).
   • tests/processors/test_extract.py: line 3 (the docstring) and line 49 (PROCESSOR_VERSION == GOLDENS["watcher_processor_version"]). That key goes away, so re-anchor the version check on co_core plus LOCAL_GENERATION.
   • tests/test_drift.py:119 names scripts/gen_parity_goldens.py. Keep it in step if the path changes.

   Traps:

   1. Prove the new generator equals the old oracle before freezing anything. Run the co-core-direct generator on 0.19.7 over cases.json. It must reproduce every digest in today's goldens.json byte for byte. If one differs, the new method isn't what Watcher computed (most likely the essence → extractor → config path, or the spec chain); find out why before going on. This is the one chance to compare against Watcher's numbers.
   2. Keep it cross-implementation. The point of AGENTS.md line 55 was that the tests can't pass by Processor agreeing with itself. The new generator imports nothing from processor (add a test that checks its imports). It uses only co-core's public pure API: extractor_for_essence, extraction_config_from_spec, canonical_text, and the spec_fingerprint used today (co_core.pure.extract.extraction_defaults, co_core.pure.extract.spec_fingerprint, per the spec's symbol list).
   3. The metadata changes shape. Record co_core, the generator's own identity, and the provenance of the frozen v1 set (watcher_commit a5d6f34…), so the history isn't lost when the next bump regenerates.
   4. Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29 and JSON: take co-core's JsonExtractor (cannobserv#354) and add a jsonpath parity fixture #37 re-entry tests change. Restate them on each issue: "a cannobserv release containing #523 / #354" alone. Drop "and Watcher ready to take it". Add "notice to Watcher, since processor_version moves" and "regenerate goldens co-core-direct and list moved digests".

   Gates:

   • Tell Watcher about the D5 change: D5 was agreed with Watcher on 2026-10-02. That's a cross-repo comment, drafted here and approved by the operator. The likely venue is the watcher issue that recorded the original agreement (the spec's D5 row names it), or watcher#350. It states decision 1 and what Watcher will be told at each bump.
   • None otherwise. The decisions are made.

   Deploy: none expected. Spec, AGENTS.md, DEPLOYMENT.md, scripts/ and tests/ aren't runtime paths, and the drift check will say "none that runs". If the generator moves into src/, that changes; don't move it there.

   Done when: #47's list; the generator reproduces today's goldens exactly (trap 1); #29 and #37 restated; the Watcher notice posted (approved); #1 closed with links.

   Ship: issue → branch in a worktree → plan (short) → TDD → PR → FF merge → git pull --ff-only → close #1. The closing comment gives each box with its evidence and the follow-ups (or "none").

   Kickoff prompt:

   Pick up processor#47. Read its hand-off comment from epic #24 first.
2. gregoryfoster

   added 7 commits that reference this issue on Oct 10, 2026

   #47 docs: CR 1 — the parity inputs are frozen with the v1 record; nev…

   ...

   4feb9de

   #47 docs: CR 2 — v1's reproduction is dated history; the standing tes…

   ...

   aabb401

   #47 docs: CR 3 — the generator's docstring names Watcher's registry d…

   ...

   93491a1

   #47 fix: CR 4 — a case whose extractor raises is named, and nothing i…

   ...

   d4bae35

   #47 docs: CR 5 — D5's row dates Watcher's pin

   ...

   aa8ae45

   #47 docs: CR 7 — spec §7's corpus list goes back on its parent bullet

   ...

   ec1c560

   #47 docs: CR 8 — spec §5's lockstep-era facts are dated to the cutover

   ...

   ab3ce5b
3. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Shipped in #49 (FF to main, 2088d71..ab3ce5b). No deploy: nothing that runs changed (processor.drift.counts is false on every path; live stays 2088d71).

   Box	Evidence
   Spec §6 step 5 marked done	linked to watcher#326's gate verdict and soak verdict, and to watcher#350; step 4 marked done too
   D5 restated as decision 1, history kept	spec D5 row, §5, an amendment entry for 2026-10-10
   AGENTS.md lines 21 and 55	rewritten; "never regenerate to make a test pass" kept
   DEPLOYMENT.md bump procedure	regenerate co-core-direct, list moved digests, notify Watcher, coordinate only on a contract change
   Generator	scripts/gen_parity_goldens.py replaced in place (same path, so tests/test_drift.py is unchanged). It imports only the stdlib and co_core.pure.extract, and an AST test enforces that
   Trap 1	on 0.19.7 the new generator reproduced all 13 of Watcher's goldens byte for byte (b1e6369, before goldens.json was touched). It stays as a standing test: the committed goldens equal the generator's output on the pin
   Trap 3, metadata	co_core, generated_by, and v1_provenance (watcher_commit a5d6f34…, 0.19.7+1), which every run carries forward
   test_extract.py re-anchored	PROCESSOR_VERSION == f"{co_core}+{LOCAL_GENERATION}"
   #29 and #37 restated	#29, #37. #37 also flags that unknown_essence_is_html uses application/json and will move at that bump
   Two CR rounds	CR 1–8: one fix (a case whose extractor raises is named, and nothing is written) and seven doc fixes

   Open: the Watcher notice of decision 1. It is drafted and awaiting the operator's approval and venue. This issue stays open until it is posted.
   Follow-ups: none.
4. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   The Watcher notice is posted (approved by the operator): watcher#325, next to the original D5 agreement. It states decision 1 and what Watcher gets at each bump: notice when processor_version moves, and coordination before a contract change.

   Every box is done: #49 shipped (see above); #29 and #37 restated; the notice posted; #1 closed with evidence. Follow-ups: none. Closing.
5. gregoryfoster

   closed this as completedon Oct 10, 2026