From b1e63691ea47549d8bcc29d17e9e8059eb53b730 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 21:48:17 +0000 Subject: [PATCH 1/9] #47 test: goldens from co-core's pure extract API, reproducing Watcher's v1 byte for byte scripts/gen_parity_goldens.py, rewritten in place: imports only the stdlib and co_core.pure.extract (a test walks its AST), and follows Watcher's a5d6f34 path. On co-core 0.19.7 it reproduces all 13 committed goldens byte for byte (trap 1), and that is now a standing test. goldens.json: digests untouched; Watcher's metadata moved, once by hand, into v1_provenance, which every run carries forward. Each run prints what moved, for the bump note. The version check is re-anchored on co_core + LOCAL_GENERATION. Co-Authored-By: Claude Opus 5.5 --- docs/plans/2026-10-10-47-post-cutover-d5.md | 47 +++++++ scripts/gen_parity_goldens.py | 132 ++++++++++++-------- tests/fixtures/parity/goldens.json | 11 +- tests/processors/test_extract.py | 30 +++-- tests/test_gen_parity_goldens.py | 105 ++++++++++++++++ tests/test_pin.py | 10 +- 6 files changed, 267 insertions(+), 68 deletions(-) create mode 100644 docs/plans/2026-10-10-47-post-cutover-d5.md create mode 100644 tests/test_gen_parity_goldens.py diff --git a/docs/plans/2026-10-10-47-post-cutover-d5.md b/docs/plans/2026-10-10-47-post-cutover-d5.md new file mode 100644 index 0000000..e189deb --- /dev/null +++ b/docs/plans/2026-10-10-47-post-cutover-d5.md @@ -0,0 +1,47 @@ +--- +title: Close out the cutover; D5 after watcher#350 (#47) +date: 2026-10-10 +status: per #47 and its hand-off from epic #24 (2026-10-10); decisions made by the operator +--- + +# Post-cutover D5 + +## Problem + +watcher#350 deleted Watcher's extraction. Two rules no longer guard anything: "co-core in lockstep with Watcher" (D5, AGENTS.md line 21) and "goldens come from Watcher's own code" (AGENTS.md line 55). `scripts/gen_parity_goldens.py` imports three Watcher symbols that #350 deleted. + +## Decisions (operator, 2026-10-10) + +1. **D5: Processor owns the co-core pin.** It coordinates with Watcher only when a bump changes the contract (the `content.process` / `content.derived` models, `canonical_text` semantics, `resolve_dispatch_essence`). It gives Watcher notice whenever `processor_version` moves. +2. **Goldens come from co-core directly.** Today's `goldens.json` is frozen as the v1 record. At a bump, regenerate from co-core's pure extract API, never through `processors/extract.py`, and list every moved digest in the bump note. + +## Approach + +**Rewrite `scripts/gen_parity_goldens.py` in place**, so the path stays the same and `tests/test_drift.py` needs no change. It runs in Processor's own venv and imports only the stdlib and `co_core.pure.extract.*`: `extractor_for_essence`, `extraction_config_from_spec`, `extraction_overrides_for_essence`, `canonical_text`, `canonical_text_fingerprint`, `spec_fingerprint`, `spec_schema_version`. It follows the path Watcher took at `a5d6f34`: +- config is `{**from_spec, **overrides}`; +- `content_fingerprint` is the sha256 of `canonical_text`, also when that is empty; +- a `spec_fingerprint` that raises `ValueError` becomes `None`. + +**Trap 1 is a standing test.** On the pinned co-core, the generator's output equals the committed `goldens` byte for byte. On 0.19.7 it reproduces Watcher's v1 numbers, and after a bump it catches a hand-edited golden. + +**Cross-implementation (trap 2).** A test walks the script's AST: nothing from `processor` or `src`, and every third-party import is under `co_core.pure.extract`. + +**Metadata (trap 3).** The new shape is `co_core`, `generated_by` (the generator's identity) and `v1_provenance`, which holds the old `generated_by`, `watcher_commit`, `watcher_processor_version` and `co_core`, plus a note that the numbers were reproduced co-core-direct on 2026-10-10. The generator carries `v1_provenance` forward on every run, and refuses to run without it. The v1 block is moved in once by hand, and the digests are not touched. `watcher_processor_version` leaves the top level. The test that read it is re-anchored on `PROCESSOR_VERSION == f"{GOLDENS['co_core']}+{LOCAL_GENERATION}"`. + +**The bump note.** The generator prints every moved digest (case, field, old → new), along with added and removed cases, so that the note can be pasted from its output. + +**Docs.** +- Spec: D5's row, the §5 bump bullet, §6 step 5 marked done (watcher#326's gate verdict 2026-10-06 and soak verdict 2026-10-09; watcher#350), §7's golden source, and an amendment entry. +- AGENTS.md: lines 21 and 55. Keep "never regenerate to pass". +- DEPLOYMENT.md: the co-core bump procedure. +- `tests/test_pin.py`, `tests/processors/test_extract.py` and `processors/extract.py`: docstrings and comments. + +**Off-repo.** +- #29 and #37 get their re-entry tests restated. +- A Watcher notice of decision 1 is drafted and posted only on approval. +- #1 is closed after the merge. + +## Not changed + +- The real corpus (`real/`, #16). +- `src/`: untouched, so there is no deploy. The drift check counts any `src/` edit as code that runs, comments included. The comments in `extract.py` still hold as history: the v1 numbers are Watcher's, and `LOCAL_GENERATION` matched Watcher's. diff --git a/scripts/gen_parity_goldens.py b/scripts/gen_parity_goldens.py index 36131f3..1a20486 100644 --- a/scripts/gen_parity_goldens.py +++ b/scripts/gen_parity_goldens.py @@ -1,66 +1,100 @@ -"""Generate the parity goldens by running **Watcher's own** extraction path. - -Parity must be cross-implementation, not self-referential: Processor's tests assert -its pure core reproduces what Watcher's local pipeline computes for the same bytes, -spec and essence (spec §7). This script therefore imports nothing from Processor. It -runs inside a Watcher checkout, whose ``src`` package it drives exactly as Watcher's -``process_watched_item`` does (essence → ``ServiceRegistry.get_extractor`` + -``extraction_overrides_for_essence`` → ``_extract_and_fingerprint``). - -Watcher's ``pipeline`` imports procrastinate, hence psycopg; ``psycopg-binary`` -supplies libpq without touching the host. No database is contacted. - - cd # its .wheelhouse holding the pinned co-core wheels - uv sync --frozen - uv run --frozen --with 'psycopg-binary>=3,<4' \\ - python /scripts/gen_parity_goldens.py /tests/fixtures/parity +"""Generate the parity goldens from **co-core's pure extract API, directly** (#47). + +Parity must not be self-referential: Processor's tests assert its pure core +(``processors/extract.py``) reproduces what co-core's extractors compute for the same +bytes, spec and essence (spec §7). This script therefore imports nothing from +Processor, only the stdlib and ``co_core.pure.extract``, and spells the path itself. +That path is the one Watcher took at ``a5d6f34``: essence → ``extractor_for_essence`` with +``{**extraction_config_from_spec, **extraction_overrides_for_essence}`` → +``canonical_text``. Since watcher#350 nothing else extracts, so co-core is the oracle. + +The goldens v1 were Watcher's (``_extract_and_fingerprint``, Watcher ``a5d6f34``, +co-core 0.19.7). This generator reproduced every one byte for byte. ``v1_provenance`` +records that, and every run carries it forward. Run it at a co-core bump only, never to +make a failing parity test pass. It prints each moved digest for the bump note. + + uv run python scripts/gen_parity_goldens.py tests/fixtures/parity """ import json -import subprocess import sys from importlib.metadata import version from pathlib import Path -from src.core.media_type import extraction_overrides_for_essence -from src.core.registry import ServiceRegistry -from src.core.validators import extraction_generation -from src.workers.pipeline import _extract_and_fingerprint +from co_core.pure.extract.canonical import canonical_text, canonical_text_fingerprint +from co_core.pure.extract.dispatch import extractor_for_essence +from co_core.pure.extract.extraction_defaults import extraction_config_from_spec +from co_core.pure.extract.media_type import extraction_overrides_for_essence +from co_core.pure.extract.spec_fingerprint import spec_fingerprint, spec_schema_version + +GENERATED_BY = "scripts/gen_parity_goldens.py (co-core's pure extract API, direct; #47)" + + +def golden(raw: bytes, essence: str | None, spec: dict) -> dict: + """One case's golden: the text's digest and size, and the spec's identity.""" + config = {**extraction_config_from_spec(spec), **extraction_overrides_for_essence(essence)} + chunks = extractor_for_essence(essence).extract(raw, config=config).chunks + try: + spec_fp = spec_fingerprint(spec) + except ValueError: # SpecFingerprintError: a diagnostic, reported as None + spec_fp = None + return { + # Empty text still fingerprints (sha256 of b""), as Watcher's did. + "content_fingerprint": canonical_text_fingerprint(chunks), + "content_size_bytes": len(canonical_text(chunks)), + "spec_fingerprint": spec_fp, + "spec_schema_version": spec_schema_version(spec), + } -def main(corpus: Path) -> None: - """Write ``corpus/goldens.json`` from Watcher's extraction of every case.""" +def goldens(corpus: Path) -> dict[str, dict]: + """Every case in ``corpus/cases.json``, by id.""" cases = json.loads((corpus / "cases.json").read_text()) - registry = ServiceRegistry() - goldens: dict[str, dict] = {} - for case in cases: - raw = (corpus / "inputs" / case["input"]).read_bytes() - essence = case["media_type"] - outcome = _extract_and_fingerprint( - raw, - [case["source_spec"]], - extractor=registry.get_extractor(essence), - extra_config=extraction_overrides_for_essence(essence), + return { + case["id"]: golden( + (corpus / "inputs" / case["input"]).read_bytes(), + case["media_type"], + case["source_spec"], ) - goldens[case["id"]] = { - "content_fingerprint": outcome.content_fingerprint, - "content_size_bytes": outcome.content_size_bytes, - "spec_fingerprint": outcome.spec_fingerprint, - "spec_schema_version": outcome.schema_version, - } - watcher_commit = subprocess.run( - ["git", "rev-parse", "HEAD"], capture_output=True, text=True, check=True - ).stdout.strip() + for case in cases + } + + +def moved(old: dict[str, dict], new: dict[str, dict]) -> list[str]: + """Each field that moved, each case removed or added: the bump note's list.""" + lines = [] + for case_id in sorted(old.keys() | new.keys()): + if case_id not in new: + lines.append(f"{case_id}: removed") + elif case_id not in old: + lines.append(f"{case_id}: added") + else: + for field in sorted(old[case_id].keys() | new[case_id].keys()): + before, after = old[case_id].get(field), new[case_id].get(field) + if before != after: + lines.append(f"{case_id} {field}: {before} -> {after}") + return lines + + +def main(corpus: Path) -> list[str]: + """Rewrite ``corpus/goldens.json`` on the installed co-core; return what moved.""" + path = corpus / "goldens.json" + previous = json.loads(path.read_text()) + if "v1_provenance" not in previous: + raise SystemExit(f"{path} has no v1_provenance: refusing to lose the v1 record") + new = goldens(corpus) doc = { - "generated_by": "scripts/gen_parity_goldens.py (Watcher's _extract_and_fingerprint)", - "watcher_commit": watcher_commit, "co_core": version("co-core"), - "watcher_processor_version": extraction_generation(), - "goldens": goldens, + "generated_by": GENERATED_BY, + "goldens": new, + "v1_provenance": previous["v1_provenance"], } - (corpus / "goldens.json").write_text(json.dumps(doc, indent=2, sort_keys=True) + "\n") - print(f"{len(goldens)} goldens from watcher@{watcher_commit[:7]} on co-core {doc['co_core']}") + path.write_text(json.dumps(doc, indent=2, sort_keys=True) + "\n") + return moved(previous["goldens"], new) if __name__ == "__main__": - main(Path(sys.argv[1]).resolve()) + changes = main(Path(sys.argv[1]).resolve()) + print(f"co-core {version('co-core')}: {len(changes)} moved") + for line in changes: + print(f" {line}") diff --git a/tests/fixtures/parity/goldens.json b/tests/fixtures/parity/goldens.json index 345f864..aad370b 100644 --- a/tests/fixtures/parity/goldens.json +++ b/tests/fixtures/parity/goldens.json @@ -1,6 +1,6 @@ { "co_core": "0.19.7", - "generated_by": "scripts/gen_parity_goldens.py (Watcher's _extract_and_fingerprint)", + "generated_by": "scripts/gen_parity_goldens.py (co-core's pure extract API, direct; #47)", "goldens": { "csv": { "content_fingerprint": "sha256:3aa128d2928d8d694df0f015dcffc146ee14d09e7dc12cc7434148b3af64e5d0", @@ -81,6 +81,11 @@ "spec_schema_version": 1 } }, - "watcher_commit": "a5d6f3417f4b559449a3f5d82f5c40438a4cc18d", - "watcher_processor_version": "0.19.7+1" + "v1_provenance": { + "co_core": "0.19.7", + "generated_by": "scripts/gen_parity_goldens.py (Watcher's _extract_and_fingerprint)", + "reproduced": "2026-10-10, #47: the co-core-direct generator matched every v1 golden byte for byte on co-core 0.19.7", + "watcher_commit": "a5d6f3417f4b559449a3f5d82f5c40438a4cc18d", + "watcher_processor_version": "0.19.7+1" + } } diff --git a/tests/processors/test_extract.py b/tests/processors/test_extract.py index 288d0d5..27ae253 100644 --- a/tests/processors/test_extract.py +++ b/tests/processors/test_extract.py @@ -1,8 +1,11 @@ -"""The pure ``extract`` core against goldens computed by Watcher's own pipeline (spec §7). - -``tests/fixtures/parity/goldens.json`` is produced by ``scripts/gen_parity_goldens.py`` -running Watcher's ``_extract_and_fingerprint`` over the same bytes, spec and essence, -so a pass here is cross-implementation parity, not a self-check. +"""The pure ``extract`` core against goldens computed outside Processor (spec §7). + +``tests/fixtures/parity/goldens.json`` is produced by ``scripts/gen_parity_goldens.py``. +That script drives co-core's pure extract API directly, over the same bytes, spec and +essence, and imports nothing from Processor. So a pass here is cross-implementation +parity, not a self-check. The v1 set was Watcher's (``_extract_and_fingerprint`` at +``a5d6f34``), and the generator reproduced it byte for byte (#47). The real corpus is +Watcher's recorded fingerprints. """ import builtins @@ -16,7 +19,12 @@ import pytest from processor.processors import TRANSFORMS -from processor.processors.extract import PROCESSOR_VERSION, ExtractOutcome, extract +from processor.processors.extract import ( + LOCAL_GENERATION, + PROCESSOR_VERSION, + ExtractOutcome, + extract, +) CORPUS = Path(__file__).resolve().parent.parent / "fixtures" / "parity" CASES = json.loads((CORPUS / "cases.json").read_text()) @@ -37,7 +45,7 @@ def _run(case: dict) -> ExtractOutcome: def test_goldens_were_generated_on_the_pinned_co_core() -> None: - # A co-core bump must regenerate the goldens from Watcher on the new version. + # A co-core bump must regenerate the goldens, co-core-direct, on the new version. assert GOLDENS["co_core"] == version("co-core") @@ -45,19 +53,19 @@ def test_every_case_has_a_golden() -> None: assert {c["id"] for c in CASES} == set(GOLDENS["goldens"]) -def test_processor_version_matches_watchers() -> None: - assert PROCESSOR_VERSION == GOLDENS["watcher_processor_version"] +def test_processor_version_is_the_goldens_co_core_plus_generation() -> None: + assert PROCESSOR_VERSION == f"{GOLDENS['co_core']}+{LOCAL_GENERATION}" @pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES]) -def test_parity_with_watcher(case: dict) -> None: +def test_parity_with_goldens(case: dict) -> None: golden = GOLDENS["goldens"][case["id"]] outcome = _run(case) assert outcome.empty is (golden["content_size_bytes"] == 0) assert len(outcome.text) == golden["content_size_bytes"] if outcome.empty: - # Watcher fingerprints b"" and then refuses it; the fact carries no digest. + # The golden fingerprints b"" (Watcher's did, then refused it); the fact carries none. assert golden["content_fingerprint"] == EMPTY_SHA256 assert outcome.text == b"" assert outcome.output_digest is None diff --git a/tests/test_gen_parity_goldens.py b/tests/test_gen_parity_goldens.py new file mode 100644 index 0000000..a2c2fac --- /dev/null +++ b/tests/test_gen_parity_goldens.py @@ -0,0 +1,105 @@ +"""``scripts/gen_parity_goldens.py``: the goldens, from co-core's pure extract API (#47). + +Since watcher#350 nothing else extracts, so the oracle is co-core itself, driven by a +script that shares no code with Processor. On the pinned co-core the generator must +reproduce the committed goldens byte for byte. On 0.19.7 those are Watcher's v1 +numbers; after a bump, they are what the bump note listed. +""" + +import ast +import importlib.util +import json +import shutil +import sys +from importlib.metadata import version +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parent.parent +SCRIPT = ROOT / "scripts" / "gen_parity_goldens.py" +CORPUS = ROOT / "tests" / "fixtures" / "parity" +GOLDENS = json.loads((CORPUS / "goldens.json").read_text()) +_spec = importlib.util.spec_from_file_location("gen_parity_goldens", SCRIPT) +gen = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(gen) + + +def _imported_modules() -> set[str]: + tree = ast.parse(SCRIPT.read_text()) + names: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + names.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, "relative import" + names.add(node.module) + return names + + +def test_imports_nothing_but_the_stdlib_and_co_cores_pure_extract() -> None: + # Cross-implementation: the goldens can't pass by Processor agreeing with itself. + for name in _imported_modules(): + root = name.split(".")[0] + if root in sys.stdlib_module_names: + continue + assert name.startswith("co_core.pure.extract"), name + + +def test_reproduces_every_committed_golden() -> None: + # Trap 1 (#47): on 0.19.7 this is Watcher's a5d6f34 output, byte for byte. + assert gen.goldens(CORPUS) == GOLDENS["goldens"] + + +def test_metadata_names_co_core_the_generator_and_the_v1_record() -> None: + assert GOLDENS["co_core"] == version("co-core") + assert GOLDENS["generated_by"] == gen.GENERATED_BY + v1 = GOLDENS["v1_provenance"] + assert v1["watcher_commit"] == "a5d6f3417f4b559449a3f5d82f5c40438a4cc18d" + assert v1["watcher_processor_version"] == "0.19.7+1" + assert v1["co_core"] == "0.19.7" + assert "watcher_processor_version" not in GOLDENS + + +@pytest.fixture +def corpus(tmp_path: Path) -> Path: + copy = tmp_path / "parity" + shutil.copytree(CORPUS / "inputs", copy / "inputs") + shutil.copy(CORPUS / "cases.json", copy) + shutil.copy(CORPUS / "goldens.json", copy) + return copy + + +def test_regenerating_on_the_pin_moves_nothing_and_keeps_the_file(corpus: Path) -> None: + assert gen.main(corpus) == [] + assert (corpus / "goldens.json").read_text() == (CORPUS / "goldens.json").read_text() + + +def test_lists_every_moved_digest(corpus: Path) -> None: + doc = json.loads((corpus / "goldens.json").read_text()) + doc["goldens"]["csv"]["content_fingerprint"] = "sha256:old" + doc["goldens"]["csv"]["content_size_bytes"] = 1 + doc["goldens"]["gone"] = doc["goldens"].pop("xlsx") + (corpus / "goldens.json").write_text(json.dumps(doc)) + + moved = gen.main(corpus) + + real = GOLDENS["goldens"]["csv"] + assert moved == [ + f"csv content_fingerprint: sha256:old -> {real['content_fingerprint']}", + f"csv content_size_bytes: 1 -> {real['content_size_bytes']}", + "gone: removed", + "xlsx: added", + ] + rewritten = json.loads((corpus / "goldens.json").read_text()) + assert rewritten["goldens"] == GOLDENS["goldens"] + assert rewritten["v1_provenance"] == GOLDENS["v1_provenance"] + + +def test_refuses_to_drop_the_v1_record(corpus: Path) -> None: + doc = json.loads((corpus / "goldens.json").read_text()) + del doc["v1_provenance"] + (corpus / "goldens.json").write_text(json.dumps(doc)) + + with pytest.raises(SystemExit, match="v1_provenance"): + gen.main(corpus) diff --git a/tests/test_pin.py b/tests/test_pin.py index efd4005..89cf4ab 100644 --- a/tests/test_pin.py +++ b/tests/test_pin.py @@ -1,8 +1,8 @@ -"""The co-core pin is exact and in lockstep with Watcher (spec D5, §5). +"""The co-core pin is exact, and Processor owns it (spec D5, amended 2026-10-10; §5). -A bump changes ``processor_version`` on every fact Watcher compares, so it must be -a deliberate act: edit ``EXPECTED`` here, in the same commit as the pin, after the -parity corpus passes on the new version. +A bump changes ``processor_version`` on every fact, and Watcher re-baselines on it, so +it must be a deliberate act: edit ``EXPECTED`` here, in the same commit as the pin, +with the goldens regenerated co-core-direct and every moved digest in the bump note. """ import tomllib @@ -41,5 +41,5 @@ def test_installed_matches_the_pin(name: str) -> None: def test_processor_version_is_co_core_plus_generation() -> None: - assert LOCAL_GENERATION == 1 # matches Watcher's LOCAL_EXTRACTION_GENERATION + assert LOCAL_GENERATION == 1 # Watcher's LOCAL_EXTRACTION_GENERATION at the cutover assert PROCESSOR_VERSION == f"{EXPECTED}+1" From e9ef37fa72aad648cb3065a4e00ee03ef863d3af Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 21:52:49 +0000 Subject: [PATCH 2/9] #47 docs: the cutover is done; D5 restated: Processor owns the pin; goldens from co-core Spec: an amendment entry; D5's row restated with its history; section 5's bump rule; section 6 steps 4 and 5 marked done (watcher#326's gate and soak verdicts, watcher#350); section 7's golden source. AGENTS.md lines 21 and 55 and DEPLOYMENT.md's bump procedure follow, with the never-regenerate-to-pass rule kept. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 +- docs/DEPLOYMENT.md | 8 ++- .../2026-09-29-processor-service-design.md | 61 +++++++++++++++---- 3 files changed, 57 insertions(+), 16 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e7ffb1b..13678a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,7 +18,7 @@ TDD required. Red → Green → Refactor. No production code without a failing t Python 3.12, uv, hatchling src layout, pytest, ruff. -**co-core is pinned exactly (`==0.19.7`), in lockstep with Watcher** (spec D5, §5). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core; a bump is planned with Watcher and the parity corpus must pass unchanged. +**co-core is pinned exactly (`==0.19.7`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core). **Cannobserv wheelhouse.** `co-core`, `co-core-aio`, `co-core-sync` resolve from `./.wheelhouse` (git-ignored) via `[tool.uv] find-links`. Populate it before any `uv` command: @@ -52,7 +52,7 @@ find-links locks by filename, not hash, so either source satisfies the same `uv. **Extraction runs in a killable child process** (`python -I -m processor._child`: timeout, `RLIMIT_AS`, `oom_score_adj` 1000, scrubbed env, allowlisting unpickler for its result), never a thread. The child is untrusted: it parses untrusted documents. **It contains itself** (`processor._contain`, #2, spec §3): Landlock (read-only on an allowlist derived from `sys.path`, no TCP, scoped signals) and seccomp (no socket at all). `processor run` is undumpable, and refuses to start where the child can't be contained (`CO_PROCESSOR_CHILD_CONTAINMENT=required`, the default). Keep the child's imports inside the allowlist: a new extractor that reads a data file outside it fails under containment, and the whole-corpus test in `tests/test_child.py` catches that. The test suite runs `required` where the kernel allows it (the pytest header says which), and never skips containment on `co-processor`. -**Parity goldens** come from Watcher's own code (`scripts/gen_parity_goldens.py`), never from Processor's. A failing parity test means output diverged from Watcher; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note. +**Parity goldens** come from co-core's pure extract API directly (`scripts/gen_parity_goldens.py`), never from Processor's code. The script imports nothing from `processor`, and a test checks that. The v1 set was Watcher's, and the generator reproduces it byte for byte (`v1_provenance`, #47). Regenerate only at a co-core bump, and list every moved digest in the bump note. A failing parity test means the output diverged; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note. ## Agent Skills diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 50649ea..d124d1a 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -453,11 +453,13 @@ On boot, the service preflights both buckets and exits non-zero if either is unr ## Bumping co-core -co-core is pinned `==` in lockstep with Watcher (spec §5). A bump changes `processor_version` on every fact, so it is planned with Watcher: +co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-10-10, #47). Until watcher#350 it was in lockstep with Watcher. A bump changes `processor_version` on every fact, and Watcher re-baselines on that: 1. Rebuild the wheelhouse at the new tag, then update the pin in `pyproject.toml` and `EXPECTED` in `tests/test_pin.py`, and `uv lock`. -2. Regenerate the goldens from Watcher on the new version (`scripts/gen_parity_goldens.py`). They must not change, or the bump note says why output moved. -3. Deploy together with Watcher's bump, or neither moves. +2. Regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. +3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved. +4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models, `canonical_text` semantics and `resolve_dispatch_essence`. +5. Merge, then `scripts/deploy.sh`. ## Tailscale DNS diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 2098551..044e9d5 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -43,6 +43,16 @@ on `co-processor`. - A command dead-lettered at the cap can get a failure fact, then a success; the failure stands (Section 4, processor#17). +**Amended 2026-10-10 (the cutover closed and watcher#350, processor#47):** +- The cutover is done (Section 6, step 5). Watcher deleted its local extraction + in watcher#350, so Processor is the only extractor. +- D5 is restated: Processor owns the co-core pin. It coordinates with Watcher only + on a contract change, and gives notice whenever `processor_version` moves + (Section 5). +- The goldens come from co-core's pure extract API directly. Watcher's v1 set is + frozen as the record, and the new generator reproduces it byte for byte + (Section 7). + --- ## Why a new service @@ -84,7 +94,7 @@ state and UI it should not have). | D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). | | D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. | | D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. | -| D5 | **co-core pinned exactly, matched to Watcher** (0.19.7 today); bumps are deliberate and coordinated. Watcher agreed on 2026-10-02 (watcher#325): it pins `==0.19.7` when #325 lands, and neither side deploys a bump until the parity corpus passes on it unchanged. | +| D5 | **co-core pinned exactly; Processor owns the pin** (0.19.7 today). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins `==0.19.7` for contracts and the bus, and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | | D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. | | D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). | | D8 | **v1 scope is #629's requirements only.** | @@ -405,12 +415,22 @@ reclaim re-running (not re-publishing) the extraction. - This section named 0.19.4 until the 2026-09-29 amendment. That pin would have reported `"0.19.4+1"` against Watcher's `"0.19.7+1"`, which is Open Question 1's hazard from the first command. -- **Bumps are deliberate and coordinated.** `processor_version` moves on every - co-core release whether or not output moves, and Watcher's Option A reacts to - it. No automatic lock refresh; dependency bots skip co-core. A bump is planned - with Watcher: the golden-digest corpus passes unchanged on the new version - before deploy, and the bump note says whether output moved. During the shadow - window both repos move together or neither does. +- **Bumps are deliberate, and Processor decides them** (D5, amended 2026-10-10). + `processor_version` moves on every co-core release whether or not output + moves. No automatic lock refresh; dependency bots skip co-core. A bump: + - regenerates the goldens from co-core's pure extract API directly + (`scripts/gen_parity_goldens.py`, never through `processors/extract.py`). + Every moved digest is listed in the bump note, as a deliberate re-baseline; + - passes the real corpus (`real/`) unchanged, or the bump note says which + fingerprints moved; + - **gives Watcher notice**, because `processor_version` moves. Watcher's + Option A re-baselines silently on a processor change alone and audits it as + `check.rebaselined`; + - **is coordinated with Watcher** only when it changes the contract: the + `content.process` / `content.derived` models, `canonical_text` semantics, + or `resolve_dispatch_essence`. +- Until watcher#350 (2026-10-10) both sides extracted, so the pin was in + lockstep with Watcher's, and both repos moved together or neither did. ## Section 6 — cross-repo changes and cutover @@ -441,9 +461,17 @@ Status re-checked against each repo on 2026-10-05 (#23). the production output store, and the digest equal to Watcher's recorded fingerprint. 4. Watcher #325 ships and runs **shadow**: Processor stores texts; Watcher's - comparator counts mismatches. + comparator counts mismatches. **Done:** watcher#325, closed 2026-10-03. 5. Watcher #326 switches on zero mismatches across a window with at least one - real change event — Watcher's gate, unchanged. + real change event — Watcher's gate, unchanged. **Done:** + - [the gate verdict](https://github.com/CannObserv/watcher/issues/326#issuecomment-6026345941): 87 of 87 shadow + comparisons matched, 2026-10-03T18:39Z → 2026-10-06T21:07Z, including one + real change. The switch to `processor` happened at 2026-10-06T21:35:24Z; + - [the soak verdict](https://github.com/CannObserv/watcher/issues/326#issuecomment-6090059341): the 3-day soak + passed, and watcher#326 closed 2026-10-09T22:07Z; + - [watcher#350](https://github.com/CannObserv/watcher/issues/350) (closed 2026-10-10T01:52Z) deleted Watcher's local + extraction, `WATCHER_EXTRACT_MODE`, the shadow leg and its + `co-core[extract]` extra. Processor is the only extractor. **Hard ordering:** `processor.process` exists before Watcher's first command. A group created from `$` afterwards skips earlier entries — Watcher's reaper @@ -456,7 +484,17 @@ TDD, red first. - **Pure core — golden-digest parity.** A corpus of HTML / PDF / CSV fixtures, each with a spec and resolved `media_type`, asserting `output_digest` and - `empty`. Include: a one-page scanned PDF (one empty chunk → `empty`, not a + `empty`. + - **The golden source (amended 2026-10-10, processor#47).** The v1 goldens + were Watcher's: `_extract_and_fingerprint` at Watcher `a5d6f34`, on co-core + 0.19.7. watcher#350 deleted that code. `scripts/gen_parity_goldens.py` now + drives co-core's pure extract API directly (`extractor_for_essence`, + `extraction_config_from_spec`, `extraction_overrides_for_essence`, + `canonical_text`, `spec_fingerprint`), and imports nothing from Processor, + which a test checks. On 0.19.7 it reproduces every v1 golden byte for byte, + also a standing test. `goldens.json` keeps the v1 provenance (`v1_provenance`) + across regenerations. It is regenerated only at a bump, never to make a + failing parity test pass. Include: a one-page scanned PDF (one empty chunk → `empty`, not a failure); an unknown essence (→ HTML); a spec whose `spec_fingerprint` raises (→ `None`). - Watcher's only file fixture is `tests/fixtures/sample.html`, so the corpus @@ -511,7 +549,8 @@ TDD, red first. - **Integration.** A scratch Redis on `co-processor` with the real stream names; a test issuer `XADD`s real commands; assert facts, stored bytes, and `XINFO GROUPS` lag, against a scratch GCS prefix or the local store. -- **Production acceptance** is Watcher's shadow mismatch count. +- **Production acceptance** was Watcher's shadow mismatch count: 87 of 87 + matched, and the cutover followed (Section 6, step 5). ## Section 8 — standup and handoff From 4feb9ded2a3f9e7ea5a7f2a48863289a68780550 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:03:36 +0000 Subject: [PATCH 3/9] =?UTF-8?q?#47=20docs:=20CR=201=20=E2=80=94=20the=20pa?= =?UTF-8?q?rity=20inputs=20are=20frozen=20with=20the=20v1=20record;=20neve?= =?UTF-8?q?r=20rerun=20their=20writer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- scripts/make_parity_inputs.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/scripts/make_parity_inputs.py b/scripts/make_parity_inputs.py index 1af5eb5..d8b8cf4 100644 --- a/scripts/make_parity_inputs.py +++ b/scripts/make_parity_inputs.py @@ -1,10 +1,13 @@ """Write the synthetic parity-corpus inputs to tests/fixtures/parity/inputs/. -Run once; the outputs are committed, and the goldens are derived from the committed -bytes (scripts/gen_parity_goldens.py). Regenerating changes the xlsx bytes (its zip -carries timestamps), so regenerate the goldens with it. - - uv run python scripts/make_parity_inputs.py +Ran once, for v1. The outputs are committed, and the goldens are derived from the +committed bytes (scripts/gen_parity_goldens.py). **Do not rerun it.** The inputs are +frozen with the goldens' v1 record. Rerunning rewrites the xlsx bytes, because its zip +carries timestamps, which would move a v1 golden outside a co-core bump (#47). Add a +new input as its own committed file, then regenerate the goldens at the bump that +needs it. + + uv run python scripts/make_parity_inputs.py # how the v1 inputs were written """ import io From aabb401c537a447d56c793b1e75a0597d03f5d2c Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:03:46 +0000 Subject: [PATCH 4/9] =?UTF-8?q?#47=20docs:=20CR=202=20=E2=80=94=20v1's=20r?= =?UTF-8?q?eproduction=20is=20dated=20history;=20the=20standing=20test=20i?= =?UTF-8?q?s=20generator=20=3D=3D=20committed=20goldens?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/specs/2026-09-29-processor-service-design.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 13678a5..8c7c0e5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ find-links locks by filename, not hash, so either source satisfies the same `uv. **Extraction runs in a killable child process** (`python -I -m processor._child`: timeout, `RLIMIT_AS`, `oom_score_adj` 1000, scrubbed env, allowlisting unpickler for its result), never a thread. The child is untrusted: it parses untrusted documents. **It contains itself** (`processor._contain`, #2, spec §3): Landlock (read-only on an allowlist derived from `sys.path`, no TCP, scoped signals) and seccomp (no socket at all). `processor run` is undumpable, and refuses to start where the child can't be contained (`CO_PROCESSOR_CHILD_CONTAINMENT=required`, the default). Keep the child's imports inside the allowlist: a new extractor that reads a data file outside it fails under containment, and the whole-corpus test in `tests/test_child.py` catches that. The test suite runs `required` where the kernel allows it (the pytest header says which), and never skips containment on `co-processor`. -**Parity goldens** come from co-core's pure extract API directly (`scripts/gen_parity_goldens.py`), never from Processor's code. The script imports nothing from `processor`, and a test checks that. The v1 set was Watcher's, and the generator reproduces it byte for byte (`v1_provenance`, #47). Regenerate only at a co-core bump, and list every moved digest in the bump note. A failing parity test means the output diverged; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note. +**Parity goldens** come from co-core's pure extract API directly (`scripts/gen_parity_goldens.py`), never from Processor's code. The script imports nothing from `processor`, and a test checks that. The v1 set was Watcher's, and the generator reproduced it byte for byte on 0.19.7 (2026-10-10, #47; `v1_provenance`). A test holds the committed goldens equal to the generator's output on the pinned co-core. Regenerate only at a co-core bump, and list every moved digest in the bump note. A failing parity test means the output diverged; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note. ## Agent Skills diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 044e9d5..9e9cc31 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -491,9 +491,11 @@ TDD, red first. drives co-core's pure extract API directly (`extractor_for_essence`, `extraction_config_from_spec`, `extraction_overrides_for_essence`, `canonical_text`, `spec_fingerprint`), and imports nothing from Processor, - which a test checks. On 0.19.7 it reproduces every v1 golden byte for byte, - also a standing test. `goldens.json` keeps the v1 provenance (`v1_provenance`) - across regenerations. It is regenerated only at a bump, never to make a + which a test checks. On 0.19.7 it reproduced every v1 golden byte for byte + (2026-10-10). A standing test holds the committed goldens equal to its output + on the pinned co-core, so after a bump they are the bump's re-baseline, not + v1. `goldens.json` keeps the v1 provenance (`v1_provenance`) across + regenerations. It is regenerated only at a bump, never to make a failing parity test pass. Include: a one-page scanned PDF (one empty chunk → `empty`, not a failure); an unknown essence (→ HTML); a spec whose `spec_fingerprint` raises (→ `None`). From 93491a1b9fd86e89485654f833200933014ef80a Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:03:59 +0000 Subject: [PATCH 5/9] =?UTF-8?q?#47=20docs:=20CR=203=20=E2=80=94=20the=20ge?= =?UTF-8?q?nerator's=20docstring=20names=20Watcher's=20registry=20dispatch?= =?UTF-8?q?,=20not=20co-core's?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- scripts/gen_parity_goldens.py | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/scripts/gen_parity_goldens.py b/scripts/gen_parity_goldens.py index 1a20486..b330ca6 100644 --- a/scripts/gen_parity_goldens.py +++ b/scripts/gen_parity_goldens.py @@ -3,15 +3,18 @@ Parity must not be self-referential: Processor's tests assert its pure core (``processors/extract.py``) reproduces what co-core's extractors compute for the same bytes, spec and essence (spec §7). This script therefore imports nothing from -Processor, only the stdlib and ``co_core.pure.extract``, and spells the path itself. -That path is the one Watcher took at ``a5d6f34``: essence → ``extractor_for_essence`` with -``{**extraction_config_from_spec, **extraction_overrides_for_essence}`` → -``canonical_text``. Since watcher#350 nothing else extracts, so co-core is the oracle. - -The goldens v1 were Watcher's (``_extract_and_fingerprint``, Watcher ``a5d6f34``, -co-core 0.19.7). This generator reproduced every one byte for byte. ``v1_provenance`` -records that, and every run carries it forward. Run it at a co-core bump only, never to -make a failing parity test pass. It prints each moved digest for the bump note. +Processor, only the stdlib and ``co_core.pure.extract``, and spells the path itself: +essence → ``extractor_for_essence`` with ``{**extraction_config_from_spec, +**extraction_overrides_for_essence}`` → ``canonical_text``. That is Watcher's path at +``a5d6f34``, except that Watcher dispatched through its ``ServiceRegistry``, whose map +was entry for entry co-core's ``EXTRACTOR_BY_ESSENCE`` (spec Open Question 3). Since +watcher#350 nothing else extracts, so co-core is the oracle. + +The v1 goldens were Watcher's (``_extract_and_fingerprint``, Watcher ``a5d6f34``, +co-core 0.19.7). On 2026-10-10 this generator reproduced every one byte for byte. +``v1_provenance`` records that, and every run carries it forward. Run it at a co-core +bump only, never to make a failing parity test pass. It prints each moved digest for +the bump note. uv run python scripts/gen_parity_goldens.py tests/fixtures/parity """ From d4bae35b0495e3c7542fab20164ac9578ab5de16 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:04:19 +0000 Subject: [PATCH 6/9] =?UTF-8?q?#47=20fix:=20CR=204=20=E2=80=94=20a=20case?= =?UTF-8?q?=20whose=20extractor=20raises=20is=20named,=20and=20nothing=20i?= =?UTF-8?q?s=20written?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit At a bump such as #29's (cannobserv#523 refuses xpath/regex/jsonpath), the traceback named only the extractor. Co-Authored-By: Claude Opus 5.5 --- scripts/gen_parity_goldens.py | 18 +++++++++--------- tests/test_gen_parity_goldens.py | 21 +++++++++++++++++++++ 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/scripts/gen_parity_goldens.py b/scripts/gen_parity_goldens.py index b330ca6..b303a58 100644 --- a/scripts/gen_parity_goldens.py +++ b/scripts/gen_parity_goldens.py @@ -51,16 +51,16 @@ def golden(raw: bytes, essence: str | None, spec: dict) -> dict: def goldens(corpus: Path) -> dict[str, dict]: - """Every case in ``corpus/cases.json``, by id.""" + """Every case in ``corpus/cases.json``, by id. A case that raises is named.""" cases = json.loads((corpus / "cases.json").read_text()) - return { - case["id"]: golden( - (corpus / "inputs" / case["input"]).read_bytes(), - case["media_type"], - case["source_spec"], - ) - for case in cases - } + out: dict[str, dict] = {} + for case in cases: + raw = (corpus / "inputs" / case["input"]).read_bytes() + try: + out[case["id"]] = golden(raw, case["media_type"], case["source_spec"]) + except Exception as exc: + raise RuntimeError(f"case {case['id']!r}: {exc}") from exc + return out def moved(old: dict[str, dict], new: dict[str, dict]) -> list[str]: diff --git a/tests/test_gen_parity_goldens.py b/tests/test_gen_parity_goldens.py index a2c2fac..a905780 100644 --- a/tests/test_gen_parity_goldens.py +++ b/tests/test_gen_parity_goldens.py @@ -103,3 +103,24 @@ def test_refuses_to_drop_the_v1_record(corpus: Path) -> None: with pytest.raises(SystemExit, match="v1_provenance"): gen.main(corpus) + + +def test_an_extractor_that_raises_names_its_case_and_writes_nothing(corpus: Path) -> None: + cases = json.loads((corpus / "cases.json").read_text()) + (corpus / "inputs" / "broken.pdf").write_bytes(b"this is not a pdf") + cases.append( + { + "id": "broken", + "input": "broken.pdf", + "media_type": "application/pdf", + "source_spec": {"extraction": {"algorithm": "full_page"}}, + } + ) + (corpus / "cases.json").write_text(json.dumps(cases)) + before = (corpus / "goldens.json").read_text() + + with pytest.raises(RuntimeError, match="case 'broken'") as raised: + gen.main(corpus) + + assert raised.value.__cause__ is not None + assert (corpus / "goldens.json").read_text() == before From aa8ae45d88023798e25bdfa884c676ee10499034 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:04:23 +0000 Subject: [PATCH 7/9] =?UTF-8?q?#47=20docs:=20CR=205=20=E2=80=94=20D5's=20r?= =?UTF-8?q?ow=20dates=20Watcher's=20pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/specs/2026-09-29-processor-service-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 9e9cc31..a6be098 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -94,7 +94,7 @@ state and UI it should not have). | D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). | | D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. | | D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. | -| D5 | **co-core pinned exactly; Processor owns the pin** (0.19.7 today). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins `==0.19.7` for contracts and the bus, and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | +| D5 | **co-core pinned exactly; Processor owns the pin** (0.19.7 today). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | | D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. | | D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). | | D8 | **v1 scope is #629's requirements only.** | From ec1c560c651a69197c3bbb5890a89223f8b78adc Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:12:19 +0000 Subject: [PATCH 8/9] =?UTF-8?q?#47=20docs:=20CR=207=20=E2=80=94=20spec=20?= =?UTF-8?q?=C2=A77's=20corpus=20list=20goes=20back=20on=20its=20parent=20b?= =?UTF-8?q?ullet?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The golden-source sub-bullet was spliced in mid-paragraph, leaving the parent's Include: sentence trailing it. Co-Authored-By: Claude Opus 5.5 --- docs/specs/2026-09-29-processor-service-design.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index a6be098..25d20f1 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -484,7 +484,9 @@ TDD, red first. - **Pure core — golden-digest parity.** A corpus of HTML / PDF / CSV fixtures, each with a spec and resolved `media_type`, asserting `output_digest` and - `empty`. + `empty`. Include: a one-page scanned PDF (one empty chunk → `empty`, not a + failure); an unknown essence (→ HTML); a spec whose `spec_fingerprint` raises + (→ `None`). - **The golden source (amended 2026-10-10, processor#47).** The v1 goldens were Watcher's: `_extract_and_fingerprint` at Watcher `a5d6f34`, on co-core 0.19.7. watcher#350 deleted that code. `scripts/gen_parity_goldens.py` now @@ -496,9 +498,7 @@ TDD, red first. on the pinned co-core, so after a bump they are the bump's re-baseline, not v1. `goldens.json` keeps the v1 provenance (`v1_provenance`) across regenerations. It is regenerated only at a bump, never to make a - failing parity test pass. Include: a one-page scanned PDF (one empty chunk → `empty`, not a - failure); an unknown essence (→ HTML); a spec whose `spec_fingerprint` raises - (→ `None`). + failing parity test pass. - Watcher's only file fixture is `tests/fixtures/sample.html`, so the corpus starts synthetic. - Before shadow it adds real samples: per watched item, the raw digest, From ab3ce5b90d7b396eee235c92bd51146a75c44d19 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Sat, 10 Oct 2026 22:12:30 +0000 Subject: [PATCH 9/9] =?UTF-8?q?#47=20docs:=20CR=208=20=E2=80=94=20spec=20?= =?UTF-8?q?=C2=A75's=20lockstep-era=20facts=20are=20dated=20to=20the=20cut?= =?UTF-8?q?over?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/specs/2026-09-29-processor-service-design.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 25d20f1..a1ee296 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -399,14 +399,16 @@ reclaim re-running (not re-publishing) the extraction. - `processor_version = processor_version(LOCAL_GENERATION)` (co-core, `co_core.pure.extract.canonical`), spelled `"+"`. - **`LOCAL_GENERATION = 1`**, matching Watcher's `LOCAL_EXTRACTION_GENERATION = - 1` — the dispatch Processor runs is Watcher's, lifted into co-core. Bump it by + **`LOCAL_GENERATION = 1`**, which matched Watcher's `LOCAL_EXTRACTION_GENERATION = + 1` at the cutover: the dispatch Processor runs is Watcher's, lifted into + co-core. Since watcher#350 nothing on Watcher's side has to match. Bump it by hand only when Processor's own logic (config merging, dispatch) changes output in a way co-core's version cannot see. -- **co-core `==0.19.7`**, the version Watcher's `uv.lock` resolves. Watcher's - `pyproject.toml` allows `>=0.19.6,<0.20`. Both sides then report `"0.19.7+1"` - from the first shadow command. Processor needs `GcsBlobStore` (0.19.3+) and - the `content.process` types (0.19.4). +- **co-core `==0.19.7`**, the version Watcher's `uv.lock` resolved at the + cutover (Watcher's `pyproject.toml` allowed `>=0.19.6,<0.20`), so both sides + reported `"0.19.7+1"` from the first shadow command. Since D5's 2026-10-10 + amendment, Watcher's lock no longer constrains this pin. Processor needs + `GcsBlobStore` (0.19.3+) and the `content.process` types (0.19.4). - Between 0.19.4 and 0.19.7, `co_core.pure.extract` and `co_core_aio.bus` are byte-identical, and the `content.process` types did not change. 0.19.6 only added the `content.persist` pair.