#47: close out the cutover; D5 restated: Processor owns the pin; goldens from co-core directly

AGENTS.md

@@ -18,7 +18,7 @@ TDD required. Red → Green → Refactor. No production code without a failing t
 
 Python 3.12, uv, hatchling src layout, pytest, ruff.
 
-**co-core is pinned exactly (`==0.19.7`), in lockstep with Watcher** (spec D5, §5). `processor_version` = `"<co-core version>+<LOCAL_GENERATION>"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core; a bump is planned with Watcher and the parity corpus must pass unchanged.
+**co-core is pinned exactly (`==0.19.7`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"<co-core version>+<LOCAL_GENERATION>"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core).
 
 **Cannobserv wheelhouse.** `co-core`, `co-core-aio`, `co-core-sync` resolve from `./.wheelhouse` (git-ignored) via `[tool.uv] find-links`. Populate it before any `uv` command:
 
@@ -52,7 +52,7 @@ find-links locks by filename, not hash, so either source satisfies the same `uv.
 
 **Extraction runs in a killable child process** (`python -I -m processor._child`: timeout, `RLIMIT_AS`, `oom_score_adj` 1000, scrubbed env, allowlisting unpickler for its result), never a thread. The child is untrusted: it parses untrusted documents. **It contains itself** (`processor._contain`, #2, spec §3): Landlock (read-only on an allowlist derived from `sys.path`, no TCP, scoped signals) and seccomp (no socket at all). `processor run` is undumpable, and refuses to start where the child can't be contained (`CO_PROCESSOR_CHILD_CONTAINMENT=required`, the default). Keep the child's imports inside the allowlist: a new extractor that reads a data file outside it fails under containment, and the whole-corpus test in `tests/test_child.py` catches that. The test suite runs `required` where the kernel allows it (the pytest header says which), and never skips containment on `co-processor`.
 
-**Parity goldens** come from Watcher's own code (`scripts/gen_parity_goldens.py`), never from Processor's. A failing parity test means output diverged from Watcher; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note.
+**Parity goldens** come from co-core's pure extract API directly (`scripts/gen_parity_goldens.py`), never from Processor's code. The script imports nothing from `processor`, and a test checks that. The v1 set was Watcher's, and the generator reproduced it byte for byte on 0.19.7 (2026-10-10, #47; `v1_provenance`). A test holds the committed goldens equal to the generator's output on the pinned co-core. Regenerate only at a co-core bump, and list every moved digest in the bump note. A failing parity test means the output diverged; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note.
 
 ## Agent Skills
 

docs/DEPLOYMENT.md

@@ -453,11 +453,13 @@ On boot, the service preflights both buckets and exits non-zero if either is unr
 
 ## Bumping co-core
 
-co-core is pinned `==` in lockstep with Watcher (spec §5). A bump changes `processor_version` on every fact, so it is planned with Watcher:
+co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-10-10, #47). Until watcher#350 it was in lockstep with Watcher. A bump changes `processor_version` on every fact, and Watcher re-baselines on that:
 
 1. Rebuild the wheelhouse at the new tag, then update the pin in `pyproject.toml` and `EXPECTED` in `tests/test_pin.py`, and `uv lock`.
-2. Regenerate the goldens from Watcher on the new version (`scripts/gen_parity_goldens.py`). They must not change, or the bump note says why output moved.
-3. Deploy together with Watcher's bump, or neither moves.
+2. Regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump.
+3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved.
+4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models, `canonical_text` semantics and `resolve_dispatch_essence`.
+5. Merge, then `scripts/deploy.sh`.
 
 ## Tailscale DNS
 

docs/plans/2026-10-10-47-post-cutover-d5.md

@@ -0,0 +1,47 @@
+---
+title: Close out the cutover; D5 after watcher#350 (#47)
+date: 2026-10-10
+status: per #47 and its hand-off from epic #24 (2026-10-10); decisions made by the operator
+---
+
+# Post-cutover D5
+
+## Problem
+
+watcher#350 deleted Watcher's extraction. Two rules no longer guard anything: "co-core in lockstep with Watcher" (D5, AGENTS.md line 21) and "goldens come from Watcher's own code" (AGENTS.md line 55). `scripts/gen_parity_goldens.py` imports three Watcher symbols that #350 deleted.
+
+## Decisions (operator, 2026-10-10)
+
+1. **D5: Processor owns the co-core pin.** It coordinates with Watcher only when a bump changes the contract (the `content.process` / `content.derived` models, `canonical_text` semantics, `resolve_dispatch_essence`). It gives Watcher notice whenever `processor_version` moves.
+2. **Goldens come from co-core directly.** Today's `goldens.json` is frozen as the v1 record. At a bump, regenerate from co-core's pure extract API, never through `processors/extract.py`, and list every moved digest in the bump note.
+
+## Approach
+
+**Rewrite `scripts/gen_parity_goldens.py` in place**, so the path stays the same and `tests/test_drift.py` needs no change. It runs in Processor's own venv and imports only the stdlib and `co_core.pure.extract.*`: `extractor_for_essence`, `extraction_config_from_spec`, `extraction_overrides_for_essence`, `canonical_text`, `canonical_text_fingerprint`, `spec_fingerprint`, `spec_schema_version`. It follows the path Watcher took at `a5d6f34`:
+- config is `{**from_spec, **overrides}`;
+- `content_fingerprint` is the sha256 of `canonical_text`, also when that is empty;
+- a `spec_fingerprint` that raises `ValueError` becomes `None`.
+
+**Trap 1 is a standing test.** On the pinned co-core, the generator's output equals the committed `goldens` byte for byte. On 0.19.7 it reproduces Watcher's v1 numbers, and after a bump it catches a hand-edited golden.
+
+**Cross-implementation (trap 2).** A test walks the script's AST: nothing from `processor` or `src`, and every third-party import is under `co_core.pure.extract`.
+
+**Metadata (trap 3).** The new shape is `co_core`, `generated_by` (the generator's identity) and `v1_provenance`, which holds the old `generated_by`, `watcher_commit`, `watcher_processor_version` and `co_core`, plus a note that the numbers were reproduced co-core-direct on 2026-10-10. The generator carries `v1_provenance` forward on every run, and refuses to run without it. The v1 block is moved in once by hand, and the digests are not touched. `watcher_processor_version` leaves the top level. The test that read it is re-anchored on `PROCESSOR_VERSION == f"{GOLDENS['co_core']}+{LOCAL_GENERATION}"`.
+
+**The bump note.** The generator prints every moved digest (case, field, old → new), along with added and removed cases, so that the note can be pasted from its output.
+
+**Docs.**
+- Spec: D5's row, the §5 bump bullet, §6 step 5 marked done (watcher#326's gate verdict 2026-10-06 and soak verdict 2026-10-09; watcher#350), §7's golden source, and an amendment entry.
+- AGENTS.md: lines 21 and 55. Keep "never regenerate to pass".
+- DEPLOYMENT.md: the co-core bump procedure.
+- `tests/test_pin.py`, `tests/processors/test_extract.py` and `processors/extract.py`: docstrings and comments.
+
+**Off-repo.**
+- #29 and #37 get their re-entry tests restated.
+- A Watcher notice of decision 1 is drafted and posted only on approval.
+- #1 is closed after the merge.
+
+## Not changed
+
+- The real corpus (`real/`, #16).
+- `src/`: untouched, so there is no deploy. The drift check counts any `src/` edit as code that runs, comments included. The comments in `extract.py` still hold as history: the v1 numbers are Watcher's, and `LOCAL_GENERATION` matched Watcher's.

docs/specs/2026-09-29-processor-service-design.md

@@ -43,6 +43,16 @@ on `co-processor`.
 - A command dead-lettered at the cap can get a failure fact, then a success;
   the failure stands (Section 4, processor#17).
 
+**Amended 2026-10-10 (the cutover closed and watcher#350, processor#47):**
+- The cutover is done (Section 6, step 5). Watcher deleted its local extraction
+  in watcher#350, so Processor is the only extractor.
+- D5 is restated: Processor owns the co-core pin. It coordinates with Watcher only
+  on a contract change, and gives notice whenever `processor_version` moves
+  (Section 5).
+- The goldens come from co-core's pure extract API directly. Watcher's v1 set is
+  frozen as the record, and the new generator reproduces it byte for byte
+  (Section 7).
+
 ---
 
 ## Why a new service
@@ -84,7 +94,7 @@ state and UI it should not have).
 | D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). |
 | D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. |
 | D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. |
-| D5 | **co-core pinned exactly, matched to Watcher** (0.19.7 today); bumps are deliberate and coordinated. Watcher agreed on 2026-10-02 (watcher#325): it pins `==0.19.7` when #325 lands, and neither side deploys a bump until the parity corpus passes on it unchanged. |
+| D5 | **co-core pinned exactly; Processor owns the pin** (0.19.7 today). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* |
 | D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. |
 | D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). |
 | D8 | **v1 scope is #629's requirements only.** |
@@ -389,14 +399,16 @@ reclaim re-running (not re-publishing) the extraction.
 
 - `processor_version = processor_version(LOCAL_GENERATION)` (co-core,
   `co_core.pure.extract.canonical`), spelled `"<co-core version>+<generation>"`.
-  **`LOCAL_GENERATION = 1`**, matching Watcher's `LOCAL_EXTRACTION_GENERATION =
-  1` — the dispatch Processor runs is Watcher's, lifted into co-core. Bump it by
+  **`LOCAL_GENERATION = 1`**, which matched Watcher's `LOCAL_EXTRACTION_GENERATION =
+  1` at the cutover: the dispatch Processor runs is Watcher's, lifted into
+  co-core. Since watcher#350 nothing on Watcher's side has to match. Bump it by
   hand only when Processor's own logic (config merging, dispatch) changes output
   in a way co-core's version cannot see.
-- **co-core `==0.19.7`**, the version Watcher's `uv.lock` resolves. Watcher's
-  `pyproject.toml` allows `>=0.19.6,<0.20`. Both sides then report `"0.19.7+1"`
-  from the first shadow command. Processor needs `GcsBlobStore` (0.19.3+) and
-  the `content.process` types (0.19.4).
+- **co-core `==0.19.7`**, the version Watcher's `uv.lock` resolved at the
+  cutover (Watcher's `pyproject.toml` allowed `>=0.19.6,<0.20`), so both sides
+  reported `"0.19.7+1"` from the first shadow command. Since D5's 2026-10-10
+  amendment, Watcher's lock no longer constrains this pin. Processor needs
+  `GcsBlobStore` (0.19.3+) and the `content.process` types (0.19.4).
   - Between 0.19.4 and 0.19.7, `co_core.pure.extract` and `co_core_aio.bus` are
     byte-identical, and the `content.process` types did not change. 0.19.6 only
     added the `content.persist` pair.
@@ -405,12 +417,22 @@ reclaim re-running (not re-publishing) the extraction.
   - This section named 0.19.4 until the 2026-09-29 amendment. That pin would
     have reported `"0.19.4+1"` against Watcher's `"0.19.7+1"`, which is Open
     Question 1's hazard from the first command.
-- **Bumps are deliberate and coordinated.** `processor_version` moves on every
-  co-core release whether or not output moves, and Watcher's Option A reacts to
-  it. No automatic lock refresh; dependency bots skip co-core. A bump is planned
-  with Watcher: the golden-digest corpus passes unchanged on the new version
-  before deploy, and the bump note says whether output moved. During the shadow
-  window both repos move together or neither does.
+- **Bumps are deliberate, and Processor decides them** (D5, amended 2026-10-10).
+  `processor_version` moves on every co-core release whether or not output
+  moves. No automatic lock refresh; dependency bots skip co-core. A bump:
+  - regenerates the goldens from co-core's pure extract API directly
+    (`scripts/gen_parity_goldens.py`, never through `processors/extract.py`).
+    Every moved digest is listed in the bump note, as a deliberate re-baseline;
+  - passes the real corpus (`real/`) unchanged, or the bump note says which
+    fingerprints moved;
+  - **gives Watcher notice**, because `processor_version` moves. Watcher's
+    Option A re-baselines silently on a processor change alone and audits it as
+    `check.rebaselined`;
+  - **is coordinated with Watcher** only when it changes the contract: the
+    `content.process` / `content.derived` models, `canonical_text` semantics,
+    or `resolve_dispatch_essence`.
+- Until watcher#350 (2026-10-10) both sides extracted, so the pin was in
+  lockstep with Watcher's, and both repos moved together or neither did.
 
 ## Section 6 — cross-repo changes and cutover
 
@@ -441,9 +463,17 @@ Status re-checked against each repo on 2026-10-05 (#23).
      the production output store, and the digest equal to Watcher's recorded
      fingerprint.
 4. Watcher #325 ships and runs **shadow**: Processor stores texts; Watcher's
-   comparator counts mismatches.
+   comparator counts mismatches. **Done:** watcher#325, closed 2026-10-03.
 5. Watcher #326 switches on zero mismatches across a window with at least one
-   real change event — Watcher's gate, unchanged.
+   real change event — Watcher's gate, unchanged. **Done:**
+   - [the gate verdict](https://github.com/CannObserv/watcher/issues/326#issuecomment-6026345941): 87 of 87 shadow
+     comparisons matched, 2026-10-03T18:39Z → 2026-10-06T21:07Z, including one
+     real change. The switch to `processor` happened at 2026-10-06T21:35:24Z;
+   - [the soak verdict](https://github.com/CannObserv/watcher/issues/326#issuecomment-6090059341): the 3-day soak
+     passed, and watcher#326 closed 2026-10-09T22:07Z;
+   - [watcher#350](https://github.com/CannObserv/watcher/issues/350) (closed 2026-10-10T01:52Z) deleted Watcher's local
+     extraction, `WATCHER_EXTRACT_MODE`, the shadow leg and its
+     `co-core[extract]` extra. Processor is the only extractor.
 
 **Hard ordering:** `processor.process` exists before Watcher's first command. A
 group created from `$` afterwards skips earlier entries — Watcher's reaper
@@ -459,6 +489,18 @@ TDD, red first.
   `empty`. Include: a one-page scanned PDF (one empty chunk → `empty`, not a
   failure); an unknown essence (→ HTML); a spec whose `spec_fingerprint` raises
   (→ `None`).
+  - **The golden source (amended 2026-10-10, processor#47).** The v1 goldens
+    were Watcher's: `_extract_and_fingerprint` at Watcher `a5d6f34`, on co-core
+    0.19.7. watcher#350 deleted that code. `scripts/gen_parity_goldens.py` now
+    drives co-core's pure extract API directly (`extractor_for_essence`,
+    `extraction_config_from_spec`, `extraction_overrides_for_essence`,
+    `canonical_text`, `spec_fingerprint`), and imports nothing from Processor,
+    which a test checks. On 0.19.7 it reproduced every v1 golden byte for byte
+    (2026-10-10). A standing test holds the committed goldens equal to its output
+    on the pinned co-core, so after a bump they are the bump's re-baseline, not
+    v1. `goldens.json` keeps the v1 provenance (`v1_provenance`) across
+    regenerations. It is regenerated only at a bump, never to make a
+    failing parity test pass.
   - Watcher's only file fixture is `tests/fixtures/sample.html`, so the corpus
     starts synthetic.
   - Before shadow it adds real samples: per watched item, the raw digest,
@@ -511,7 +553,8 @@ TDD, red first.
 - **Integration.** A scratch Redis on `co-processor` with the real stream names;
   a test issuer `XADD`s real commands; assert facts, stored bytes, and `XINFO
   GROUPS` lag, against a scratch GCS prefix or the local store.
-- **Production acceptance** is Watcher's shadow mismatch count.
+- **Production acceptance** was Watcher's shadow mismatch count: 87 of 87
+  matched, and the cutover followed (Section 6, step 5).
 
 ## Section 8 — standup and handoff