From 415fa24d2f5cb6b71e4459786aae101b1c7cfc75 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 17:40:00 +0000 Subject: [PATCH 01/11] =?UTF-8?q?#29=20docs:=20plan=20=E2=80=94=20bump=20c?= =?UTF-8?q?o-core=200.19.7=20=E2=86=92=200.32.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/plans/2026-10-12-29-co-core-0.32.0.md | 88 ++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 docs/plans/2026-10-12-29-co-core-0.32.0.md diff --git a/docs/plans/2026-10-12-29-co-core-0.32.0.md b/docs/plans/2026-10-12-29-co-core-0.32.0.md new file mode 100644 index 0000000..90310a3 --- /dev/null +++ b/docs/plans/2026-10-12-29-co-core-0.32.0.md @@ -0,0 +1,88 @@ +--- +title: Bump co-core 0.19.7 → 0.32.0 — XML and feed extractors, the refusals (#29) +date: 2026-10-12 +status: per #29, its re-entry test (met by cannobserv v0.32.0) and the operator's catch-up note (2026-10-10); reviewed 2026-10-12 (subagent: every case below run through 0.32.0's chain, findings folded in) +--- + +# co-core 0.32.0 + +## Problem + +cannobserv v0.32.0 (tag `v0.32.0`, `33e781a6`) carries cannobserv#523: +- `text/xml`, `application/xml`, `application/rss+xml` and `application/atom+xml` dispatch to a new `XmlExtractor`, where they took the HTML fallback; +- a spec nothing can honour raises `UnsupportedExtractionError(ValueError)`, where it silently meant the whole page; +- `resolve_dispatch_essence` maps `.xml` / `.rss` / `.atom` URLs to the XML essences. That is the contract, so the bump is coordinate-first (D5). + +The pin is 0.19.7. Per the operator's catch-up note, whichever of #29 / #37 bumps first takes the full jump. + +## The catch-up reading (0.19.7 → 0.32.0) + +Checked against the code (`git diff v0.19.7 v0.32.0`), not only the CHANGELOG: +- **Contract models** (`co_core.pure.models.changes`): a docstring only. `extraction_error` now also covers a refused spec; the `Literal` is unchanged. +- **`canonical_text`, `spec_fingerprint`**: unchanged. +- **Bus** (`streams.py`): comments only (`processor.process` documented as live, cannobserv#503/#528). `envelope`, `dead_letter`, `exceptions`, `effects.bus`, `co_core_aio.bus`: unchanged. +- **Blob store** (`util.blobstore`, `util.hashing`, `co_core_sync.drivers.blobstore`): unchanged. +- **Extract**: `xml_feed.py` (new, lxml, already in the extra), `errors.py` (new), the dispatch map, the extension tiebreak, `extraction_config_from_spec`'s refusals, and `refuse_unhonoured` in the HTML, PDF and CSV extractors. +- **Dependencies**: none move. Each package's `pyproject.toml` changes only its version and the sibling floor (`>=0.32,<0.33`). +- **Security fixes in 0.20–0.32**: none recorded. +- Everything else in the range is the WordPress / co/v1 surface, which Processor doesn't import. + +So `processors/extract.py` needs no code change: a refusal propagates, the child reports it as `raised`, and the handler publishes `extraction_error`, terminal (spec §4, "Extractor raises"). + +## Approach + +**Pin.** `pyproject.toml` (the three packages), `EXPECTED` in `tests/test_pin.py`, `uv lock`. `LOCAL_GENERATION` stays 1 (ask 2). `scripts/build_wheelhouse.sh`'s default tag follows the pin. + +**Refusals are goldens too.** `gen_parity_goldens.py` raises on any case that raises. A case co-core *refuses* is now recorded as `{"refused": "UnsupportedExtractionError"}` (the class name, from co-core's own raise), so the refusal is co-core's output, not Processor's expectation. The catch is `UnsupportedExtractionError` only, imported from `co_core.pure.extract.errors` (inside the import test's allowance): XML and PDF parse failures are plain `ValueError`s, and still name their case and write nothing (a malformed-XML generator test pins that). `moved()` already handles differing field sets, so a spec that starts or stops being refused at a later bump shows in the note. +- `test_extract.py`: a refused golden means `extract()` raises `UnsupportedExtractionError`. The digest test skips refused cases; the no-I/O test runs them too (a refusal does no I/O either). +- `test_child.py`'s whole-corpus run: a refused case comes back `raised`, its detail starting `UnsupportedExtractionError:`, under containment. With the handler's existing `raised` → `extraction_error` row, that is the end-to-end check. + +**New cases** (`cases.json`, before regenerating, so their goldens come from co-core): + +| Case | Input | Essence | Spec | Covers | +|---|---|---|---|---| +| `feed_rss` | `tailscale_security_bulletins.xml` | `text/xml` | full_page | ask 3. co-core's own trimmed fixture, verbatim; its digest is cross-checked against co-core's hand-pinned `sha256:78e28497…` | +| `feed_rss_rss_essence` | same | `application/rss+xml` | full_page | the essence routes | +| `feed_atom` | `feed.atom` (synthetic) | `application/atom+xml` | full_page | Atom `id` / `updated` / `link rel` | +| `feed_no_items` | `empty_feed.rss` (synthetic) | `application/rss+xml` | full_page | channel fields left out → `empty` | +| `xml_document` | `register.xml` (synthetic) | `application/xml` | full_page | a non-feed root gives its text | +| `xml_xpath` | `register.xml` | `application/xml` | xpath `//entry/title` | xpath honoured on XML | +| `refused_xpath_on_html` | `agenda.html` | `text/html` | xpath `//p` | until cannobserv#557 | +| `refused_css_on_xml` | tailscale | `text/xml` | css `item` | | +| `refused_css_on_pdf` | `minutes.pdf` | `application/pdf` | css `#main` | | +| `refused_css_on_csv` | `licenses.csv` | `text/csv` | css `#main` | | +| `refused_xpath_on_csv` | `licenses.csv` | `text/csv` | xpath `//a` | | +| `refused_regex` | tailscale | `application/rss+xml` | regex | | +| `refused_jsonpath` | `items.json` | `application/json` | jsonpath | until cannobserv#354 | +| `refused_empty_xpath` | `register.xml` | `application/xml` | xpath `""` | | +| `refused_unknown_algorithm` | `agenda.html` | `text/html` | `semantic` | | +| `refused_invalid_xpath` | `register.xml` | `application/xml` | xpath `//[` | | + +The `xhtml` case (`application/xhtml+xml`) stays HTML: there is no `+xml` suffix rule. + +**Expected moves.** The 13 existing goldens should not move (the HTML, PDF and CSV paths only gained a refusal none of them trips). The real corpus (9 items, all `text/html` + `css`) should pass unchanged. + +**Live specs.** The real corpus is the 2026-10-02 export, and the upstream exposure count is from 2026-10-05, so neither proves today's specs. The journal doesn't carry specs, but it shows the same 4 info sources since 2026-10-05, 204 of 204 commands `complete`. The gate is therefore post-deploy: the first command from each of the 4 completes, and no `extraction_error` carries `UnsupportedExtractionError`. Otherwise roll back (DEPLOYMENT.md) and raise it on #29. + +**Docs.** +- AGENTS.md: line 21 (the pin), the Owns line (XML joins HTML/PDF/CSV), and the parity paragraph (refusals are goldens). +- Spec: D5 "(0.19.7 today)", the Owns line, §2's dependency row, §4's "Extractor raises" row (names the refusal), §5's pin bullet, §6's cannobserv#503 row (now released), §7's corpus bullet (refusals; XML on the synthetic corpus) and pin-test line, the contract quick reference (heading 0.19.4–0.32.0 after re-checking each row, plus an `UnsupportedExtractionError` row), Open Question 2's version range, and an amendment entry. +- `processors/extract.py`'s module docstring (the parity target is co-core now; a refusal propagates like any raise). +- DEPLOYMENT.md: the bump procedure gains "add new-essence fixtures before regenerating" and "refusals are recorded as goldens". +- Historical 0.19.7 references stay (v1 provenance, the real corpus's `0.19.7+1`). + +**Off-repo (each post needs the operator's approval).** +- **Coordinate with Watcher (watcher#366, already filed).** Either order is safe: Processor never re-resolves the essence, and each side alone keeps an ambiguous-header `.xml` URL on the HTML path, today's behaviour. Post on watcher#366 before deploy: the new `processor_version` `0.32.0+1`, the moved digests, and that ordering argument. Deploy once Watcher acknowledges, or when the operator rules that the ordering argument makes acknowledgement unnecessary. +- #29: the bump note (moved digests, real corpus, refusal check). + +## Steps + +1. Red: the cases and fixtures, the refused-golden handling in the generator test, `test_extract.py`, `test_child.py`, and `EXPECTED = "0.32.0"`. They fail on 0.19.7. +2. Green: the pin and the lock; the generator's refusal record; regenerate; paste the moved list. +3. Docs. +4. Full gate (`ruff check`, `ruff format --check`, `pytest` with the integration marker, `uv lock --locked`), PR with `Refs #29` (the live deploy and the Watcher notice remain). +5. After merge: the post on watcher#366 and its acknowledgement; `scripts/deploy.sh`; the live-spec gate above; the drift check reads current. + +## Open questions + +None blocking. The live fetch of Tailscale's feed (46 items, 146 KB) isn't added: co-core's trimmed copy is the same feed, and its digest is independently pinned upstream. From 7021b408e6009618f63c3e822cc9c56d374212db Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 17:41:16 +0000 Subject: [PATCH 02/11] #29 test: XML/feed cases, refusals as goldens, the 0.32.0 pin (red) Co-Authored-By: Claude Opus 5.5 --- tests/fixtures/parity/cases.json | 36 ++++++++- tests/fixtures/parity/inputs/empty_feed.rss | 9 +++ tests/fixtures/parity/inputs/feed.atom | 22 ++++++ tests/fixtures/parity/inputs/register.xml | 11 +++ .../inputs/tailscale_security_bulletins.xml | 78 +++++++++++++++++++ tests/processors/test_extract.py | 50 ++++++++++-- tests/test_child.py | 10 ++- tests/test_gen_parity_goldens.py | 33 ++++++++ tests/test_pin.py | 2 +- 9 files changed, 243 insertions(+), 8 deletions(-) create mode 100644 tests/fixtures/parity/inputs/empty_feed.rss create mode 100644 tests/fixtures/parity/inputs/feed.atom create mode 100644 tests/fixtures/parity/inputs/register.xml create mode 100644 tests/fixtures/parity/inputs/tailscale_security_bulletins.xml diff --git a/tests/fixtures/parity/cases.json b/tests/fixtures/parity/cases.json index ff15062..a360147 100644 --- a/tests/fixtures/parity/cases.json +++ b/tests/fixtures/parity/cases.json @@ -25,5 +25,39 @@ {"id": "spec_fingerprint_raises", "input": "agenda.html", "media_type": "text/html", "source_spec": {"extraction": {"algorithm": "full_page"}, "threshold": 0.5}}, {"id": "spec_schema_version_2", "input": "agenda.html", "media_type": "text/html", - "source_spec": {"schema_version": 2, "extraction": {"algorithm": "css", "selector": "#main"}}} + "source_spec": {"schema_version": 2, "extraction": {"algorithm": "css", "selector": "#main"}}}, + {"id": "feed_rss", "input": "tailscale_security_bulletins.xml", "media_type": "text/xml", + "source_spec": {"extraction": {"algorithm": "full_page"}}, + "source": "cannobserv v0.32.0 tests/co_core/pure/extract/fixtures/, verbatim (#29)"}, + {"id": "feed_rss_rss_essence", "input": "tailscale_security_bulletins.xml", + "media_type": "application/rss+xml", "source_spec": {"extraction": {"algorithm": "full_page"}}}, + {"id": "feed_atom", "input": "feed.atom", "media_type": "application/atom+xml", + "source_spec": {"extraction": {"algorithm": "full_page"}}}, + {"id": "feed_no_items", "input": "empty_feed.rss", "media_type": "application/rss+xml", + "source_spec": {"extraction": {"algorithm": "full_page"}}}, + {"id": "xml_document", "input": "register.xml", "media_type": "application/xml", + "source_spec": {"extraction": {"algorithm": "full_page"}}}, + {"id": "xml_xpath", "input": "register.xml", "media_type": "application/xml", + "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//entry/title"}}}, + {"id": "refused_xpath_on_html", "input": "agenda.html", "media_type": "text/html", + "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//p"}}}, + {"id": "refused_css_on_xml", "input": "tailscale_security_bulletins.xml", "media_type": "text/xml", + "source_spec": {"extraction": {"algorithm": "css", "selector": "item"}}}, + {"id": "refused_css_on_pdf", "input": "minutes.pdf", "media_type": "application/pdf", + "source_spec": {"extraction": {"algorithm": "css", "selector": "#main"}}}, + {"id": "refused_css_on_csv", "input": "licenses.csv", "media_type": "text/csv", + "source_spec": {"extraction": {"algorithm": "css", "selector": "#main"}}}, + {"id": "refused_xpath_on_csv", "input": "licenses.csv", "media_type": "text/csv", + "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//a"}}}, + {"id": "refused_regex", "input": "tailscale_security_bulletins.xml", + "media_type": "application/rss+xml", + "source_spec": {"extraction": {"algorithm": "regex", "selector": "TS-\\d+"}}}, + {"id": "refused_jsonpath", "input": "items.json", "media_type": "application/json", + "source_spec": {"extraction": {"algorithm": "jsonpath", "selector": "$.items"}}}, + {"id": "refused_empty_xpath", "input": "register.xml", "media_type": "application/xml", + "source_spec": {"extraction": {"algorithm": "xpath", "selector": ""}}}, + {"id": "refused_unknown_algorithm", "input": "agenda.html", "media_type": "text/html", + "source_spec": {"extraction": {"algorithm": "semantic"}}}, + {"id": "refused_invalid_xpath", "input": "register.xml", "media_type": "application/xml", + "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//["}}} ] diff --git a/tests/fixtures/parity/inputs/empty_feed.rss b/tests/fixtures/parity/inputs/empty_feed.rss new file mode 100644 index 0000000..694e460 --- /dev/null +++ b/tests/fixtures/parity/inputs/empty_feed.rss @@ -0,0 +1,9 @@ + + + + Rulemaking updates + https://example.org/rulemaking/ + No updates yet + Mon, 12 Oct 2026 14:39:44 GMT + + diff --git a/tests/fixtures/parity/inputs/feed.atom b/tests/fixtures/parity/inputs/feed.atom new file mode 100644 index 0000000..bc10146 --- /dev/null +++ b/tests/fixtures/parity/inputs/feed.atom @@ -0,0 +1,22 @@ + + + Board notices + urn:example:board-notices + 2026-10-12T09:00:00Z + + + urn:example:notice:42 + Special <em>meeting</em> called + + + 2026-10-11T17:30:00-07:00 + 2026-10-12T08:00:00Z + The board meets on Thursday. + + + urn:example:notice:41 + Agenda posted + + 2026-10-01T12:00:00 + + diff --git a/tests/fixtures/parity/inputs/register.xml b/tests/fixtures/parity/inputs/register.xml new file mode 100644 index 0000000..7f755f2 --- /dev/null +++ b/tests/fixtures/parity/inputs/register.xml @@ -0,0 +1,11 @@ + + + + Proposed rule: packaging and labeling + 2026-09-30 + + + Hearing notice: retail licensing + 2026-10-02 + + diff --git a/tests/fixtures/parity/inputs/tailscale_security_bulletins.xml b/tests/fixtures/parity/inputs/tailscale_security_bulletins.xml new file mode 100644 index 0000000..adfc683 --- /dev/null +++ b/tests/fixtures/parity/inputs/tailscale_security_bulletins.xml @@ -0,0 +1,78 @@ + + + + Security Bulletins on Tailscale + https://tailscale.com/security-bulletins/ + Recent security bulletins from Tailscale + Mon, 12 Oct 2026 14:39:44 GMT + https://validator.w3.org/feed/docs/rss2.html + tailscale.com + en-US + © 2026 Tailscale Inc. All rights reserved. + + + TS-2026-011 + https://tailscale.com/security-bulletins/#ts-2026-011 + https://tailscale.com/security-bulletins/#ts-2026-011 + Tue, 18 Aug 2026 00:00:00 GMT + <p><strong><em>Description</em></strong>: Insufficient validation of 4via6 subnet router destinations permitted access to host-scoped addresses, including cloud metadata and loopback services.</p> +<h4>What happened?</h4> +<p><a href="/docs/features/subnet-routers/4via6-subnets">4via6 subnet routers</a> embed an IPv4 address within an IPv6 address. When a peer sends traffic to such an address, the router unmaps the embedded IPv4 address and forwards the traffic to it.</p> +<p>Previously, this forwarding path did not check the class of the embedded IPv4 address. A peer with ACL access to an advertised 4via6 site could therefore reach host-scoped addresses as seen from the router, including its cloud metadata service and services bound to loopback. Tailscale's packet filter already drops link-local destinations, but for 4via6 traffic it only sees the outer IPv6 address, so that check previously did not run against the embedded IPv4 destination.</p> +<p>Tailscale now refuses host-scoped IPv4 destinations (loopback, link-local, multicast and broadcast) at every point that acts on an unmapped 4via6 address.</p> +<p>This vulnerability is fixed in Tailscale version 1.102.3 or newer.</p> +<h4>What was the impact?</h4> +<p>A peer with ACL access to an advertised 4via6 route could reach host-scoped addresses from the router's network namespace. On cloud-hosted routers, this included the cloud metadata service, allowing the peer to obtain the router's instance credentials. On any router, services bound to loopback became reachable, bypassing the host firewall.</p> +<h4>Who was affected?</h4> +<p>Tailnets using 4via6 subnet routing where a subnet router advertises a site prefix and a peer has ACL access to it.</p> +<h4>What do I need to do?</h4> +<p>If you use 4via6 subnet routing, upgrade your subnet routers to Tailscale version 1.102.3 or newer.</p> +<h4>Credits</h4> +<p>We would like to thank the Anthropic infrastructure security team for reporting this issue.</p> + + + + TS-2026-010 + https://tailscale.com/security-bulletins/#ts-2026-010 + https://tailscale.com/security-bulletins/#ts-2026-010 + Fri, 24 Jul 2026 00:00:00 GMT + <p><strong><em>Description</em></strong>: Tailscale SSH exposed accepted environment variable values on the command line of a local process, disclosing them to other local users and to logs.</p> +<h4>What happened?</h4> +<p><a href="/docs/features/tailscale-ssh">Tailscale SSH</a> allows the configured <code>acceptEnv</code> environment variables to be forwarded from an SSH client to the process launched on the host. Previously, these accepted variables were JSON-encoded and passed to a child process via an <code>--encoded-env</code> command line flag.</p> +<p>As a result, the names and values of the forwarded environment variables were visible in <code>/proc/&#x3C;pid>/cmdline</code> to any other local user on the host for the duration of process startup. They were also recorded in the session-start arguments that Tailscale logs locally and, unless the node was configured with <code>--no-logs-no-support</code>, to <code>log.tailscale.com</code>. Environment variables commonly carry secrets such as tokens or credentials, which could therefore be disclosed to unintended parties.</p> +<p>Tailscale SSH now passes accepted environment variables to the child process through an inherited file descriptor rather than on the command line and keeps their values out of session logs.</p> +<p>This vulnerability is fixed in Tailscale version 1.102.1 or newer.</p> +<h4>What was the impact?</h4> +<p>Secrets carried in forwarded environment variables could be disclosed to other local users on the host via <code>/proc</code>, and were written to session logs both locally and to <code>log.tailscale.com</code>.</p> +<h4>Who was affected?</h4> +<p>Users of Tailscale SSH who forward environment variables via <code>acceptEnv</code>, particularly on shared hosts with other local users.</p> +<h4>What do I need to do?</h4> +<p>If you use Tailscale SSH with <code>acceptEnv</code>, upgrade to Tailscale version 1.102.1 or newer. If you use <code>acceptEnv</code> to pass credentials, revoke and rotate any impacted credentials.</p> +<h4>Credits</h4> +<p>We would like to thank Dante Bertana for reporting this issue.</p> + + + + TS-2026-008 + https://tailscale.com/security-bulletins/#ts-2026-008 + https://tailscale.com/security-bulletins/#ts-2026-008 + Mon, 13 Jul 2026 00:00:00 GMT + <p><strong><em>Description</em></strong>: A single malformed HTTP request to a node running Tailscale Serve or Funnel could pin a CPU core indefinitely, causing denial of service.</p> +<h4>What happened?</h4> +<p><a href="/docs/features/tailscale-serve">Tailscale Serve</a> and <a href="/docs/features/tailscale-funnel">Tailscale Funnel</a> proxy incoming HTTP requests to local backends by matching the request path against the configured mount points. When resolving the handler for a request, Tailscale walked the request path upward one directory at a time, expecting the walk to eventually terminate at the root path <code>/</code>.</p> +<p>For requests whose path did not begin with <code>/</code>, this walk never reached <code>/</code> and never matched a mount point, causing the loop to spin forever. As the server enforced no request timeout, nothing interrupted the spin, and the goroutine held one core at 100% for the life of the process.</p> +<p>Tailscale now terminates the path walk for non-absolute paths, returning no handler and closing the request.</p> +<p>This vulnerability is fixed in Tailscale version 1.98.9 or newer.</p> +<h4>What was the impact?</h4> +<p>An attacker could send a crafted HTTP request to permanently consume one CPU core on the target node.</p> +<p>For Tailscale Serve, the request could originate from any peer on the tailnet with access to the node. For Tailscale Funnel, the request could originate from any unauthenticated host on the internet.</p> +<h4>Who was affected?</h4> +<p>Nodes running Tailscale Serve or Tailscale Funnel on versions prior to 1.98.9.</p> +<h4>What do I need to do?</h4> +<p>If you run Tailscale Serve or Tailscale Funnel, upgrade to Tailscale version 1.98.9 or newer.</p> +<h4>Credits</h4> +<p>We would like to thank Anthropic and Ada Logics for reporting this issue.</p> + + + + diff --git a/tests/processors/test_extract.py b/tests/processors/test_extract.py index 27ae253..c33e635 100644 --- a/tests/processors/test_extract.py +++ b/tests/processors/test_extract.py @@ -17,6 +17,7 @@ from pathlib import Path import pytest +from co_core.pure.extract.errors import UnsupportedExtractionError from processor.processors import TRANSFORMS from processor.processors.extract import ( @@ -37,6 +38,11 @@ REAL_EXPORT = json.loads((REAL / "export.json").read_text()) REAL_ITEMS = REAL_EXPORT["items"] EMPTY_SHA256 = "sha256:" + hashlib.sha256(b"").hexdigest() +REFUSED = {i for i, g in GOLDENS["goldens"].items() if "refused" in g} +EXTRACTED = [c for c in CASES if c["id"] not in REFUSED] +# co-core's own test pins this digest for the same fixture, worked out by hand from the +# feed-line contract (cannobserv v0.32.0, tests/co_core/pure/extract/test_canonical.py). +TAILSCALE_FEED_SHA256 = "sha256:78e284978faf41fbe8ccd8dc06e6d2803d9788097fca633486dfbbbdc024be09" def _run(case: dict) -> ExtractOutcome: @@ -57,7 +63,7 @@ def test_processor_version_is_the_goldens_co_core_plus_generation() -> None: assert PROCESSOR_VERSION == f"{GOLDENS['co_core']}+{LOCAL_GENERATION}" -@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES]) +@pytest.mark.parametrize("case", EXTRACTED, ids=[c["id"] for c in EXTRACTED]) def test_parity_with_goldens(case: dict) -> None: golden = GOLDENS["goldens"][case["id"]] outcome = _run(case) @@ -76,7 +82,34 @@ def test_parity_with_goldens(case: dict) -> None: assert outcome.processor_version == PROCESSOR_VERSION -@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES]) +@pytest.mark.parametrize("case", sorted(REFUSED)) +def test_parity_with_refusals(case: str) -> None: + # co-core refused the spec when the goldens were generated (#29, cannobserv#523): + # the core must raise too, never fall back to the whole page. + (spec,) = [c for c in CASES if c["id"] == case] + assert GOLDENS["goldens"][case] == {"refused": "UnsupportedExtractionError"} + with pytest.raises(UnsupportedExtractionError): + _run(spec) + + +def test_the_corpus_covers_the_refusals_and_the_xml_essences() -> None: + # The #523 surface: every XML essence dispatched, and a spec refused at each layer. + essences = {c["media_type"] for c in EXTRACTED} + assert {"text/xml", "application/xml", "application/rss+xml", "application/atom+xml"} <= ( + essences + ) + assert len(REFUSED) >= 10 + + +def test_the_feed_golden_is_co_cores_own_pin() -> None: + # Independent of the generator: co-core worked this digest out by hand. + assert GOLDENS["goldens"]["feed_rss"]["content_fingerprint"] == TAILSCALE_FEED_SHA256 + assert GOLDENS["goldens"]["feed_rss_rss_essence"]["content_fingerprint"] == ( + TAILSCALE_FEED_SHA256 + ) + + +@pytest.mark.parametrize("case", EXTRACTED, ids=[c["id"] for c in EXTRACTED]) def test_digest_covers_the_returned_bytes(case: dict) -> None: outcome = _run(case) if not outcome.empty: @@ -89,18 +122,25 @@ def test_extractor_errors_propagate() -> None: extract(b"this is not a pdf", "application/pdf", {"extraction": {}}) +def _extract_or_refuse(raw: bytes, case: dict) -> None: + try: + extract(raw, case["media_type"], case["source_spec"]) + except UnsupportedExtractionError: + assert case["id"] in REFUSED + + def test_no_io(monkeypatch: pytest.MonkeyPatch) -> None: inputs = [(CORPUS / "inputs" / c["input"]).read_bytes() for c in CASES] for case, raw in zip(CASES, inputs, strict=True): # warm lazy imports first - extract(raw, case["media_type"], case["source_spec"]) + _extract_or_refuse(raw, case) def refuse(*_a, **_k): raise AssertionError("the pure core performed I/O") monkeypatch.setattr(builtins, "open", refuse) monkeypatch.setattr(socket, "socket", refuse) - for case, raw in zip(CASES, inputs, strict=True): - extract(raw, case["media_type"], case["source_spec"]) + for case, raw in zip(CASES, inputs, strict=True): # a refusal does no I/O either + _extract_or_refuse(raw, case) def test_registry_routes_extract() -> None: diff --git a/tests/test_child.py b/tests/test_child.py index 427cb52..612637d 100644 --- a/tests/test_child.py +++ b/tests/test_child.py @@ -171,9 +171,13 @@ def _whole_corpus() -> list[tuple[str, tuple]]: async def test_the_whole_parity_corpus_is_identical_through_the_child() -> None: # cases.json and real/ (#16), under CONTAINMENT: the allowlist must cover every - # import and data file an extractor reaches for (#2). + # import and data file an extractor reaches for (#2). A spec co-core refuses comes + # back raised, which the handler publishes as extraction_error, terminal (#29). jobs = _whole_corpus() assert len(jobs) > len(json.loads((Path(TESTS) / "fixtures/parity/cases.json").read_text())) + goldens = json.loads((Path(TESTS) / "fixtures/parity/goldens.json").read_text())["goldens"] + refused = {case_id for case_id, golden in goldens.items() if "refused" in golden} + assert refused for job_id, args in jobs: result = await run_in_child( transform_target(extract), @@ -182,6 +186,10 @@ async def test_the_whole_parity_corpus_is_identical_through_the_child() -> None: rlimit_as_bytes=3 * GiB, containment=CONTAINMENT, ) + if job_id in refused: + assert result.kind == "raised", (job_id, result.detail) + assert result.detail.startswith("UnsupportedExtractionError: "), job_id + continue assert result.kind == "ok", (job_id, result.detail) assert result.value == extract(*args), job_id diff --git a/tests/test_gen_parity_goldens.py b/tests/test_gen_parity_goldens.py index a905780..fb6e1b8 100644 --- a/tests/test_gen_parity_goldens.py +++ b/tests/test_gen_parity_goldens.py @@ -105,6 +105,39 @@ def test_refuses_to_drop_the_v1_record(corpus: Path) -> None: gen.main(corpus) +def _append_case( + corpus: Path, case_id: str, name: str, body: bytes, essence: str, spec: dict +) -> None: + cases = json.loads((corpus / "cases.json").read_text()) + (corpus / "inputs" / name).write_bytes(body) + cases.append({"id": case_id, "input": name, "media_type": essence, "source_spec": spec}) + (corpus / "cases.json").write_text(json.dumps(cases)) + + +def test_a_spec_co_core_refuses_is_recorded_as_refused(corpus: Path) -> None: + # #29: the refusal is co-core's output, so it is a golden like any digest. + spec = {"extraction": {"algorithm": "regex", "selector": "x"}} + _append_case(corpus, "regex", "page.html", b"

x

", "text/html", spec) + + moved = gen.main(corpus) + + assert moved == ["regex: added"] + doc = json.loads((corpus / "goldens.json").read_text()) + assert doc["goldens"]["regex"] == {"refused": "UnsupportedExtractionError"} + + +def test_a_malformed_xml_body_is_not_a_refusal(corpus: Path) -> None: + # A parse failure is a plain ValueError: it names its case, never becomes a golden. + spec = {"extraction": {"algorithm": "full_page"}} + _append_case(corpus, "broken_xml", "broken.xml", b"", "application/xml", spec) + before = (corpus / "goldens.json").read_text() + + with pytest.raises(RuntimeError, match="case 'broken_xml'"): + gen.main(corpus) + + assert (corpus / "goldens.json").read_text() == before + + def test_an_extractor_that_raises_names_its_case_and_writes_nothing(corpus: Path) -> None: cases = json.loads((corpus / "cases.json").read_text()) (corpus / "inputs" / "broken.pdf").write_bytes(b"this is not a pdf") diff --git a/tests/test_pin.py b/tests/test_pin.py index 89cf4ab..850b486 100644 --- a/tests/test_pin.py +++ b/tests/test_pin.py @@ -13,7 +13,7 @@ from processor.processors.extract import LOCAL_GENERATION, PROCESSOR_VERSION -EXPECTED = "0.19.7" +EXPECTED = "0.32.0" ROOT = Path(__file__).resolve().parent.parent PINNED = ("co-core", "co-core-aio", "co-core-sync") From 861239e3b290d0b00eb76357fa26360f6547a1f5 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 17:44:33 +0000 Subject: [PATCH 03/11] =?UTF-8?q?#29=20feat:=20co-core=200.32.0=20?= =?UTF-8?q?=E2=80=94=20XML=20and=20feed=20extractors,=20refusals=20as=20go?= =?UTF-8?q?ldens?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pin moves 0.19.7 → 0.32.0 (processor_version 0.32.0+1; LOCAL_GENERATION stays 1). The lock moves only co-core, co-core-aio and co-core-sync. gen_parity_goldens.py records a spec co-core refuses (UnsupportedExtractionError, cannobserv#523) as {"refused": ...}; any other raise still names its case. Bump note, regenerated co-core-direct on 0.32.0: 16 moved, all added; none of the 13 existing goldens moved. feed_atom, feed_no_items, feed_rss, feed_rss_rss_essence, xml_document, xml_xpath: added refused_css_on_csv, refused_css_on_pdf, refused_css_on_xml, refused_empty_xpath, refused_invalid_xpath, refused_jsonpath, refused_regex, refused_unknown_algorithm, refused_xpath_on_csv, refused_xpath_on_html: added feed_rss is co-core's own pinned digest (sha256:78e28497…). The real corpus passes unchanged, 9/9. Co-Authored-By: Claude Opus 5.5 --- pyproject.toml | 8 ++-- scripts/build_wheelhouse.sh | 2 +- scripts/gen_parity_goldens.py | 15 +++++-- tests/fixtures/parity/goldens.json | 68 +++++++++++++++++++++++++++++- uv.lock | 18 ++++---- 5 files changed, 93 insertions(+), 18 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 7db2082..7eb4114 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,12 +5,12 @@ description = "The cohort's headless discrete-transform service: consume content readme = "AGENTS.md" requires-python = ">=3.12" dependencies = [ - # Exact, in lockstep with Watcher (spec D5, §5). processor_version is + # Exact, and Processor owns the pin (spec D5, §5). processor_version is # "+", so a bump moves every fact Watcher # compares; tests/test_pin.py makes one a deliberate, test-failing act. - "co-core[extract]==0.19.7", - "co-core-aio[bus]==0.19.7", - "co-core-sync[gcs]==0.19.7", + "co-core[extract]==0.32.0", + "co-core-aio[bus]==0.32.0", + "co-core-sync[gcs]==0.32.0", "pydantic>=2.12,<3", "pydantic-settings>=2.6,<3", # Imported directly: the consumer classifies redis-py's ResponseError / diff --git a/scripts/build_wheelhouse.sh b/scripts/build_wheelhouse.sh index 705ef10..3d330bf 100755 --- a/scripts/build_wheelhouse.sh +++ b/scripts/build_wheelhouse.sh @@ -8,7 +8,7 @@ set -euo pipefail root="$(cd "$(dirname "$0")/.." && pwd)" -tag="${1:-v0.19.7}" +tag="${1:-v0.32.0}" : "${GH_TOKEN_CANNOBSERV:?set GH_TOKEN_CANNOBSERV (read access to CannObserv/cannobserv)}" work="$(mktemp -d)" diff --git a/scripts/gen_parity_goldens.py b/scripts/gen_parity_goldens.py index b303a58..24c2c75 100644 --- a/scripts/gen_parity_goldens.py +++ b/scripts/gen_parity_goldens.py @@ -16,6 +16,10 @@ bump only, never to make a failing parity test pass. It prints each moved digest for the bump note. +A spec co-core refuses (``UnsupportedExtractionError``, cannobserv#523) is co-core's +output too, so it is recorded as ``{"refused": ""}`` (#29). Any other raise, +a parse failure included, names its case and writes nothing. + uv run python scripts/gen_parity_goldens.py tests/fixtures/parity """ @@ -26,6 +30,7 @@ from co_core.pure.extract.canonical import canonical_text, canonical_text_fingerprint from co_core.pure.extract.dispatch import extractor_for_essence +from co_core.pure.extract.errors import UnsupportedExtractionError from co_core.pure.extract.extraction_defaults import extraction_config_from_spec from co_core.pure.extract.media_type import extraction_overrides_for_essence from co_core.pure.extract.spec_fingerprint import spec_fingerprint, spec_schema_version @@ -34,9 +39,13 @@ def golden(raw: bytes, essence: str | None, spec: dict) -> dict: - """One case's golden: the text's digest and size, and the spec's identity.""" - config = {**extraction_config_from_spec(spec), **extraction_overrides_for_essence(essence)} - chunks = extractor_for_essence(essence).extract(raw, config=config).chunks + """One case's golden: the text's digest and size, and the spec's identity; or the + refusal, when co-core can't honour the spec for this essence.""" + try: + config = {**extraction_config_from_spec(spec), **extraction_overrides_for_essence(essence)} + chunks = extractor_for_essence(essence).extract(raw, config=config).chunks + except UnsupportedExtractionError as exc: + return {"refused": type(exc).__name__} try: spec_fp = spec_fingerprint(spec) except ValueError: # SpecFingerprintError: a diagnostic, reported as None diff --git a/tests/fixtures/parity/goldens.json b/tests/fixtures/parity/goldens.json index aad370b..36b8e9f 100644 --- a/tests/fixtures/parity/goldens.json +++ b/tests/fixtures/parity/goldens.json @@ -1,5 +1,5 @@ { - "co_core": "0.19.7", + "co_core": "0.32.0", "generated_by": "scripts/gen_parity_goldens.py (co-core's pure extract API, direct; #47)", "goldens": { "csv": { @@ -8,6 +8,30 @@ "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", "spec_schema_version": 1 }, + "feed_atom": { + "content_fingerprint": "sha256:ff164f9d404e71e8988386bd63283f7876d88df7e9bd7ff538e131ddc100b122", + "content_size_bytes": 184, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, + "feed_no_items": { + "content_fingerprint": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "content_size_bytes": 0, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, + "feed_rss": { + "content_fingerprint": "sha256:78e284978faf41fbe8ccd8dc06e6d2803d9788097fca633486dfbbbdc024be09", + "content_size_bytes": 422, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, + "feed_rss_rss_essence": { + "content_fingerprint": "sha256:78e284978faf41fbe8ccd8dc06e6d2803d9788097fca633486dfbbbdc024be09", + "content_size_bytes": 422, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, "html_css": { "content_fingerprint": "sha256:54fb39f5ee423006094f811bc0a36e0345187dfbbdb9935bd0425d954d6a77b6", "content_size_bytes": 142, @@ -50,6 +74,36 @@ "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", "spec_schema_version": 1 }, + "refused_css_on_csv": { + "refused": "UnsupportedExtractionError" + }, + "refused_css_on_pdf": { + "refused": "UnsupportedExtractionError" + }, + "refused_css_on_xml": { + "refused": "UnsupportedExtractionError" + }, + "refused_empty_xpath": { + "refused": "UnsupportedExtractionError" + }, + "refused_invalid_xpath": { + "refused": "UnsupportedExtractionError" + }, + "refused_jsonpath": { + "refused": "UnsupportedExtractionError" + }, + "refused_regex": { + "refused": "UnsupportedExtractionError" + }, + "refused_unknown_algorithm": { + "refused": "UnsupportedExtractionError" + }, + "refused_xpath_on_csv": { + "refused": "UnsupportedExtractionError" + }, + "refused_xpath_on_html": { + "refused": "UnsupportedExtractionError" + }, "spec_fingerprint_raises": { "content_fingerprint": "sha256:8c815816bbcc0e3ab0fb9ec8b9e77f7aa638bb2ada24cf373112ded580acaebd", "content_size_bytes": 181, @@ -79,6 +133,18 @@ "content_size_bytes": 103, "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", "spec_schema_version": 1 + }, + "xml_document": { + "content_fingerprint": "sha256:f9ecc98df3ac0ce686ba443a36834f691383569144ccc94e7d5a4ba2d62f9736", + "content_size_bytes": 92, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, + "xml_xpath": { + "content_fingerprint": "sha256:2f9f812138eb9ff74f59956bda959d30516c293b13e99b7327f170851569ae75", + "content_size_bytes": 70, + "spec_fingerprint": "spec1:sha256:d794b955037fdfca3bfa1c5e58af4d868fb5fbeb24be5940a9eda561e0bab326", + "spec_schema_version": 1 } }, "v1_provenance": { diff --git a/uv.lock b/uv.lock index 8bd57d1..ec105c5 100644 --- a/uv.lock +++ b/uv.lock @@ -268,14 +268,14 @@ wheels = [ [[package]] name = "co-core" -version = "0.19.7" +version = "0.32.0" source = { registry = ".wheelhouse" } dependencies = [ { name = "dateparser" }, { name = "pydantic" }, ] wheels = [ - { path = "co_core-0.19.7-py3-none-any.whl" }, + { path = "co_core-0.32.0-py3-none-any.whl" }, ] [package.optional-dependencies] @@ -289,14 +289,14 @@ extract = [ [[package]] name = "co-core-aio" -version = "0.19.7" +version = "0.32.0" source = { registry = ".wheelhouse" } dependencies = [ { name = "co-core" }, { name = "httpx" }, ] wheels = [ - { path = "co_core_aio-0.19.7-py3-none-any.whl" }, + { path = "co_core_aio-0.32.0-py3-none-any.whl" }, ] [package.optional-dependencies] @@ -306,7 +306,7 @@ bus = [ [[package]] name = "co-core-sync" -version = "0.19.7" +version = "0.32.0" source = { registry = ".wheelhouse" } dependencies = [ { name = "co-core" }, @@ -314,7 +314,7 @@ dependencies = [ { name = "urllib3" }, ] wheels = [ - { path = "co_core_sync-0.19.7-py3-none-any.whl" }, + { path = "co_core_sync-0.32.0-py3-none-any.whl" }, ] [package.optional-dependencies] @@ -867,9 +867,9 @@ dev = [ [package.metadata] requires-dist = [ - { name = "co-core", extras = ["extract"], specifier = "==0.19.7" }, - { name = "co-core-aio", extras = ["bus"], specifier = "==0.19.7" }, - { name = "co-core-sync", extras = ["gcs"], specifier = "==0.19.7" }, + { name = "co-core", extras = ["extract"], specifier = "==0.32.0" }, + { name = "co-core-aio", extras = ["bus"], specifier = "==0.32.0" }, + { name = "co-core-sync", extras = ["gcs"], specifier = "==0.32.0" }, { name = "google-api-core", specifier = ">=2.27,<3" }, { name = "google-auth", specifier = ">=2,<3" }, { name = "google-cloud-storage", specifier = ">=3,<4" }, From a59a38cfd79182bce68ccd3fbc5d462afbda1018 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 17:47:35 +0000 Subject: [PATCH 04/11] #29 docs: the 0.32.0 pin, XML and refusals in the spec, AGENTS.md and the bump procedure Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 6 +- docs/DEPLOYMENT.md | 2 +- .../2026-09-29-processor-service-design.md | 66 +++++++++++++------ src/processor/processors/extract.py | 7 +- 4 files changed, 55 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1828146..7a4bd52 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ The cohort's **headless discrete-transform service**. Consumes `content.process` Design of record: [docs/specs/2026-09-29-processor-service-design.md](docs/specs/2026-09-29-processor-service-design.md) (the spec). Tracking issue: #1. **Read the spec before changing behaviour**; it is self-contained and names every co-core symbol used. -**Owns:** transforms (co-core's HTML/PDF/CSV extractors now; org adapters later), `gs://co-gcs-processor`, draining `content.process.dlq`. **Does not own:** spec authoring or interpretation (issuers), fetching (Replicator), change detection/diffing (Watcher), the registry (Archiver). +**Owns:** transforms (co-core's HTML/PDF/CSV/XML extractors now; org adapters later), `gs://co-gcs-processor`, draining `content.process.dlq`. **Does not own:** spec authoring or interpretation (issuers), fetching (Replicator), change detection/diffing (Watcher), the registry (Archiver). ## Development Methodology @@ -18,7 +18,7 @@ TDD required. Red → Green → Refactor. No production code without a failing t Python 3.12, uv, hatchling src layout, pytest, ruff. -**co-core is pinned exactly (`==0.19.7`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core). +**co-core is pinned exactly (`==0.32.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core). **Cannobserv wheelhouse.** `co-core`, `co-core-aio`, `co-core-sync` resolve from `./.wheelhouse` (git-ignored) via `[tool.uv] find-links`. Populate it before any `uv` command: @@ -52,7 +52,7 @@ find-links locks by filename, not hash, so either source satisfies the same `uv. **Extraction runs in a killable child process** (`python -I -m processor._child`: timeout, `RLIMIT_AS`, `oom_score_adj` 1000, scrubbed env, allowlisting unpickler for its result), never a thread. The child is untrusted: it parses untrusted documents. **It contains itself** (`processor._contain`, #2, spec §3): Landlock (read-only on an allowlist derived from `sys.path`, no TCP, scoped signals) and seccomp (no socket at all). `processor run` is undumpable, and refuses to start where the child can't be contained (`CO_PROCESSOR_CHILD_CONTAINMENT=required`, the default). Keep the child's imports inside the allowlist: a new extractor that reads a data file outside it fails under containment, and the whole-corpus test in `tests/test_child.py` catches that. The test suite runs `required` where the kernel allows it (the pytest header says which), and never skips containment on `co-processor`. -**Parity goldens** come from co-core's pure extract API directly (`scripts/gen_parity_goldens.py`), never from Processor's code. The script imports nothing from `processor`, and a test checks that. The v1 set was Watcher's, and the generator reproduced it byte for byte on 0.19.7 (2026-10-10, #47; `v1_provenance`). A test holds the committed goldens equal to the generator's output on the pinned co-core. Regenerate only at a co-core bump, and list every moved digest in the bump note. A failing parity test means the output diverged; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note. +**Parity goldens** come from co-core's pure extract API directly (`scripts/gen_parity_goldens.py`), never from Processor's code. The script imports nothing from `processor`, and a test checks that. The v1 set was Watcher's, and the generator reproduced it byte for byte on 0.19.7 (2026-10-10, #47; `v1_provenance`). A test holds the committed goldens equal to the generator's output on the pinned co-core. A spec co-core refuses (`UnsupportedExtractionError`) is a golden too, `{"refused": …}`; any other raise is a generator error (#29). Regenerate only at a co-core bump, and list every moved digest in the bump note. A failing parity test means the output diverged; do not regenerate the goldens to make it pass. The real corpus, `tests/fixtures/parity/real/` (#16), is Watcher's export verbatim and the only copy of its blobs (gone from `gs://co-gcs-blobs` after the TTL): never edit or regenerate it; a bump that moves its output says so in the bump note. ## Agent Skills diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index ab55652..27d7810 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -456,7 +456,7 @@ On boot, the service preflights both buckets and exits non-zero if either is unr co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-10-10, #47). Until watcher#350 it was in lockstep with Watcher. A bump changes `processor_version` on every fact, and Watcher re-baselines on that: 1. Rebuild the wheelhouse at the new tag, then update the pin in `pyproject.toml` and `EXPECTED` in `tests/test_pin.py`, and `uv lock`. -2. Regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. +2. A release that adds an essence or an extraction rule gets its cases in `tests/fixtures/parity/cases.json` first, so their goldens come from co-core. Then regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved, and every case that started or stopped being refused. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. A catch-up across releases reads each CHANGELOG's Downstream lines for the extract, bus and blob-store surfaces, and diffs those packages' code between the tags (#29 did 0.19.7 → 0.32.0). 3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved. 4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models, `canonical_text` semantics and `resolve_dispatch_essence`. 5. Merge, then `scripts/deploy.sh`. diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index a1ee296..9b5648c 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -53,6 +53,15 @@ on `co-processor`. frozen as the record, and the new generator reproduces it byte for byte (Section 7). +**Amended 2026-10-12 (co-core 0.32.0, processor#29):** +- The pin moves from 0.19.7 to **0.32.0**: the catch-up jump, carrying + cannobserv#523 (Sections 2 and 5). The four XML essences dispatch to co-core's + `XmlExtractor`. A spec the dispatched extractor can't honour raises + `UnsupportedExtractionError`, a terminal `extraction_error` (Section 4). +- The bump changes `resolve_dispatch_essence`, so it was coordinated with Watcher + (watcher#366, Section 5). +- The corpus covers the XML essences, and a refusal is a golden (Section 7). + --- ## Why a new service @@ -94,7 +103,7 @@ state and UI it should not have). | D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). | | D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. | | D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. | -| D5 | **co-core pinned exactly; Processor owns the pin** (0.19.7 today). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | +| D5 | **co-core pinned exactly; Processor owns the pin** (0.32.0 today, processor#29). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | | D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. | | D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). | | D8 | **v1 scope is #629's requirements only.** | @@ -111,7 +120,7 @@ an ops concern (logs, the broker probe, lag); a cohort-wide plane is later work. function of (input bytes, spec, `processor_version`). Consumes `content.process`, publishes `content.derived`. -**Owns:** the transforms (co-core's generic HTML/PDF/CSV extractors now; org +**Owns:** the transforms (co-core's generic HTML/PDF/CSV/XML extractors now; org adapters later, selected by `command.processor` + spec); its bucket; draining `content.process.dlq` (the writer of a queue is its drainer). @@ -132,7 +141,7 @@ processor-version pin on the command; no job API — the bus is the interface. | Thing | Value | |---|---| | Repo | `CannObserv/processor` — Python 3.12, uv, hatchling src layout | -| Dependencies | `co-core[extract]`, `co-core-aio[bus]`, `co-core-sync[gcs]`, all `==0.19.7` (Section 5), from the private CannObserv index | +| Dependencies | `co-core[extract]`, `co-core-aio[bus]`, `co-core-sync[gcs]`, all `==0.32.0` (Section 5), from the private CannObserv index | | CI | wheelhouse pulled through WIF as `co-pypi-reader` (org var `GCP_WIF_PROVIDER`), as in the other cohort repos | | VM | exe.dev `co-processor`, 8 GB, default `exeuntu` image, tag `processor` | | Service | systemd unit `processor`: `MemoryMax` below VM RAM, `Restart=on-failure`, `OOMPolicy=continue` (Section 4). Runs as its own user `processor` (no sudo, not in `docker`, no home), from a read-only release at `/srv/processor/live` that `scripts/deploy.sh` builds from a commit on `origin/main` (amended 2026-10-06, #2: the cohort's release standard, broker#22 / status#9) | @@ -298,7 +307,7 @@ Observo adopted in #395/#407) so the later plane can ingest them unchanged. | Malformed digest, or `input_uri` the input store does not recognize | `invalid_input`, terminal (issuer does not re-fetch) | yes | | Input bytes gone, or the download fails its checksum (`DataCorruption`, amended 2026-09-30) | `input_unreadable`, terminal for the command (issuer re-fetches, capped) | yes | | Bytes hash ≠ `input_digest` | `input_digest_mismatch`, terminal | yes | -| Extractor raises, including `MemoryError` under `RLIMIT_AS` | `extraction_error`, terminal | yes | +| Extractor raises, including `MemoryError` under `RLIMIT_AS`, and a spec co-core refuses (`UnsupportedExtractionError` since 0.32.0, cannobserv#523: `regex`, `jsonpath`, an empty or invalid `xpath`, an unknown algorithm, `xpath` on HTML, `css` on XML / PDF / CSV), amended 2026-10-12 | `extraction_error`, terminal | yes | | Child timeout or crash | — | no; the 3rd attempt publishes `extraction_error`, terminal, and acks | | A non-transient exception escaping the handler (a bug, or an ack or dead-letter refused), amended 2026-09-30, 2026-10-02 and 2026-10-05 | at the cap, `extraction_error`, terminal, unless this entry's fact already went out (a fact published before a refused ack stands) | no; counted like a strike, and the 3rd attempt publishes, then dead-letters the entry with the exception as its reason. A dead-letter that does not land is retried alone, without running the command again (#28) | | GCS 5xx / 429 / timeout / auth; broker `NOPERM`; broker `OOM command not allowed`; broker `MISCONF` / `BUSY` / `MASTERDOWN` / `TRYAGAIN` / `CLUSTERDOWN` / `NOREPLICAS` (amended 2026-09-30) | — | no; the entry is reclaimed | @@ -404,10 +413,19 @@ reclaim re-running (not re-publishing) the extraction. co-core. Since watcher#350 nothing on Watcher's side has to match. Bump it by hand only when Processor's own logic (config merging, dispatch) changes output in a way co-core's version cannot see. -- **co-core `==0.19.7`**, the version Watcher's `uv.lock` resolved at the - cutover (Watcher's `pyproject.toml` allowed `>=0.19.6,<0.20`), so both sides - reported `"0.19.7+1"` from the first shadow command. Since D5's 2026-10-10 - amendment, Watcher's lock no longer constrains this pin. Processor needs +- **co-core `==0.32.0`** since processor#29 (2026-10-12), the catch-up jump from + 0.19.7, the version Watcher's `uv.lock` resolved at the cutover (Watcher's + `pyproject.toml` allowed `>=0.19.6,<0.20`), so both sides reported `"0.19.7+1"` + from the first shadow command. Since D5's 2026-10-10 amendment, Watcher's lock + no longer constrains this pin. + - Across 0.19.7 → 0.32.0, the bus, the stores, `canonical_text`, + `spec_fingerprint` and the `content.process` / `content.derived` models are + unchanged in code (a `changes.py` docstring and `streams.py` comments aside). + The extract package gains `XmlExtractor` and the refusals (cannobserv#523), + and `resolve_dispatch_essence` maps `.xml` / `.rss` / `.atom` to the XML + essences: a contract change, coordinated on watcher#366. Processor never + re-resolves the essence, so either side moving first keeps such a URL behind + an ambiguous header on the HTML path, as before. Processor needs `GcsBlobStore` (0.19.3+) and the `content.process` types (0.19.4). - Between 0.19.4 and 0.19.7, `co_core.pure.extract` and `co_core_aio.bus` are byte-identical, and the `content.process` types did not change. 0.19.6 only @@ -445,7 +463,7 @@ Status re-checked against each repo on 2026-10-05 (#23). | processor | Founding spec (this file, at the same path), `AGENTS.md`, #1 (from observo#629). | **Done.** | | broker | New issue: ACL user `processor` (Section 2); strip `observo`'s grants; probe watches `processor.process`; participants table and `docs/STREAMS.md`; `docs/NETWORK-PATHS.md` client cells from `co-processor`. | **Done:** broker#75, closed 2026-10-05. | | watcher | On #325 or a design-doc PR: Section 3's processor is Processor; output root `gs://co-gcs-processor`; the reader grant; the `WATCHER_EXTRACT_MODE` value `observo` (rename is Watcher's call); the Section 5 question below; the lockstep bump policy. | **Done:** watcher#325, closed 2026-10-03. | -| cannobserv | Doc-only issue: `CHANGE_BUS.md` names Observo / `observo.process` as the processor. | **Resolved upstream:** cannobserv#503 (closed 2026-10-05) names the group `processor.process` (on `main`, unreleased; the pinned 0.19.7 still carries the old docstrings). No issue filed from here. The row named the wrong file. The stale text was co-core's, not `CHANGE_BUS.md`: `streams.py` (module docstring, `CONTENT_PROCESS` comment) and `test_streams.py`. | +| cannobserv | Doc-only issue: `CHANGE_BUS.md` names Observo / `observo.process` as the processor. | **Resolved upstream:** cannobserv#503 (closed 2026-10-05) names the group `processor.process` (released in 0.24.3; the pin carries it since 0.32.0, processor#29). No issue filed from here. The row named the wrong file. The stale text was co-core's, not `CHANGE_BUS.md`: `streams.py` (module docstring, `CONTENT_PROCESS` comment) and `test_streams.py`. | | replicator | New issue amending #69's roles charter (closed): "Observo transforms" becomes Observo = stream transforms and interpretation, Processor = discrete transforms. | **Moot, no post.** The quoted "Observo transforms" is in no Replicator file. The committed charter (`docs/contracts/replicator-boundaries.md`, "The successor question left this repo") keeps Replicator out of processing, points to archiver#179 and names no home. "Leaning Observo" survives only in #69's closing comment, which points to archiver#179, so the archiver comment corrects the chain. | | archiver | Informational comment on #179: the processor is Processor. | **Done:** [archiver#179 comment](https://github.com/CannObserv/archiver/issues/179#issuecomment-6002156775), 2026-10-05. | | GCP / tailnet | Bucket, service account and grants (Section 2); the tailnet rule. Operator acts. | **Done.** GCP: #14; the tailnet rule 2026-09-29 ([DEPLOYMENT.md](../DEPLOYMENT.md) prerequisites). | @@ -484,11 +502,20 @@ from `$` on an empty stream, ahead of Watcher's first command. TDD, red first. -- **Pure core — golden-digest parity.** A corpus of HTML / PDF / CSV fixtures, - each with a spec and resolved `media_type`, asserting `output_digest` and - `empty`. Include: a one-page scanned PDF (one empty chunk → `empty`, not a - failure); an unknown essence (→ HTML); a spec whose `spec_fingerprint` raises - (→ `None`). +- **Pure core — golden-digest parity.** A corpus of HTML / PDF / CSV / XML + fixtures, each with a spec and resolved `media_type`, asserting `output_digest` + and `empty`, or the refusal. Include: a one-page scanned PDF (one empty chunk → + `empty`, not a failure); an unknown essence (→ HTML); a spec whose + `spec_fingerprint` raises (→ `None`). + - **XML and refusals (amended 2026-10-12, processor#29).** Each XML essence: + an RSS feed (co-core's own trimmed copy of Tailscale's security bulletins, + whose digest co-core pins by hand), an Atom feed, a feed with no items + (`empty`), a generic document, and an `xpath`. A spec co-core refuses is + recorded by the generator as `{"refused": "UnsupportedExtractionError"}`; + the core must raise it, and through the child it comes back `raised`. One + refusal at each layer: `extraction_config_from_spec` (`regex`, `jsonpath`, + an empty `xpath`, an unknown algorithm) and each extractor (`xpath` on HTML + and CSV; `css` on XML, PDF and CSV; an invalid `xpath` on XML). - **The golden source (amended 2026-10-10, processor#47).** The v1 goldens were Watcher's: `_extract_and_fingerprint` at Watcher `a5d6f34`, on co-core 0.19.7. watcher#350 deleted that code. `scripts/gen_parity_goldens.py` now @@ -510,12 +537,12 @@ TDD, red first. `tests/fixtures/parity/real/` keeps the export verbatim and the raw blobs, copied before Replicator's 7-day TTL. Processor reproduces every recorded fingerprint and `spec_fingerprint`, 9/9, including two revisions from before - fetch commands existed and one recorded under `0.19.4+1`. PDF, CSV and XLSX - still rest on the synthetic corpus. + fetch commands existed and one recorded under `0.19.4+1`. PDF, CSV, XLSX and + XML still rest on the synthetic corpus. - Parity with Watcher's local path should be proven before shadow, not discovered in it. - **Pin test.** The lock's co-core equals the expected exact version and - `processor_version(LOCAL_GENERATION) == "0.19.7+1"` — a bump is a deliberate, + `processor_version(LOCAL_GENERATION) == "0.32.0+1"` — a bump is a deliberate, test-failing act. - **Shells — one test per Section 4 row.** Input/output stores: a local co-core store on a temp dir behind the same interface, plus fakes raising GCS 5xx / @@ -587,7 +614,7 @@ writing-plans for v1 there. Implementation happens on `co-processor`. --- -## Contract quick reference (co-core 0.19.4–0.19.7, unchanged across them) +## Contract quick reference (co-core 0.19.4–0.32.0, unchanged across them but the refusal, added in 0.32.0) | Need | Name | Module | |---|---|---| @@ -596,6 +623,7 @@ writing-plans for v1 there. Implementation happens on `co-processor`. | Command / facts | `ContentProcessCommand`, `ProcessingCompleteEmit`, `ProcessingFailedEmit`, `ProcessingFailureReason` | `co_core.pure.models.changes` | | Dispatch | `extractor_for_essence` | `co_core.pure.extract.dispatch` (needs the `extract` extra) | | Config | `extraction_config_from_spec` | `co_core.pure.extract.extraction_defaults` | +| Refusal (0.32.0+) | `UnsupportedExtractionError` (a `ValueError`) | `co_core.pure.extract.errors` | | Essence overrides | `extraction_overrides_for_essence` | `co_core.pure.extract.media_type` | | Canonical bytes | `canonical_text`, `canonical_text_fingerprint`, `CANONICAL_TEXT_MEDIA_TYPE`, `processor_version` | `co_core.pure.extract.canonical` | | Spec identity | `spec_fingerprint`, `spec_schema_version` | `co_core.pure.extract.spec_fingerprint` | @@ -627,7 +655,7 @@ echoed, reporting only. `ChangeRevision.processor_version` is never rewritten. `spec_fingerprint` gets the same treatment. 2. ~~**co-core-aio:** does `AsyncBusConsumer` expose a delivery count or a - max-deliveries hook?~~ **Answered, no (0.19.4 through 0.19.7).** It offers + max-deliveries hook?~~ **Answered, no (0.19.4 through 0.32.0).** It offers primitives only: `ensure_group`, `read`, `ack`, `claim_stale` / `claim_stale_page`, and `dead_letter`. It has no delivery count, no hook and no run loop. The in-memory counter stays, and Processor owns its loop. diff --git a/src/processor/processors/extract.py b/src/processor/processors/extract.py index 516cef6..e4402ad 100644 --- a/src/processor/processors/extract.py +++ b/src/processor/processors/extract.py @@ -1,8 +1,9 @@ """The ``extract`` transform: raw bytes + one ``source_spec`` → canonical text (spec §3). -Pure: no I/O. The golden-digest parity target against Watcher's local extraction -(``tests/fixtures/parity``). Extractor exceptions propagate unclassified; the child -runner reports them as ``extraction_error``. +Pure: no I/O. The golden-digest parity target against co-core's pure extract API +(``tests/fixtures/parity``, #47). Extractor exceptions propagate unclassified, a spec +co-core refuses (``UnsupportedExtractionError``, #29) among them; the child runner +reports them as ``extraction_error``. """ from dataclasses import dataclass From 9210b92726f80ccde770b2b04ea9c39cc1066d6b Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:10:50 +0000 Subject: [PATCH 05/11] =?UTF-8?q?#29=20docs:=20CR=201=20=E2=80=94=20spec?= =?UTF-8?q?=20=C2=A75's=20floor=20sentence=20goes=20back=20on=20its=20pare?= =?UTF-8?q?nt=20bullet?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/specs/2026-09-29-processor-service-design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 9b5648c..3693668 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -417,7 +417,8 @@ reclaim re-running (not re-publishing) the extraction. 0.19.7, the version Watcher's `uv.lock` resolved at the cutover (Watcher's `pyproject.toml` allowed `>=0.19.6,<0.20`), so both sides reported `"0.19.7+1"` from the first shadow command. Since D5's 2026-10-10 amendment, Watcher's lock - no longer constrains this pin. + no longer constrains this pin. Processor needs `GcsBlobStore` (0.19.3+) and the + `content.process` types (0.19.4). - Across 0.19.7 → 0.32.0, the bus, the stores, `canonical_text`, `spec_fingerprint` and the `content.process` / `content.derived` models are unchanged in code (a `changes.py` docstring and `streams.py` comments aside). @@ -425,8 +426,7 @@ reclaim re-running (not re-publishing) the extraction. and `resolve_dispatch_essence` maps `.xml` / `.rss` / `.atom` to the XML essences: a contract change, coordinated on watcher#366. Processor never re-resolves the essence, so either side moving first keeps such a URL behind - an ambiguous header on the HTML path, as before. Processor needs - `GcsBlobStore` (0.19.3+) and the `content.process` types (0.19.4). + an ambiguous header on the HTML path, as before. - Between 0.19.4 and 0.19.7, `co_core.pure.extract` and `co_core_aio.bus` are byte-identical, and the `content.process` types did not change. 0.19.6 only added the `content.persist` pair. From 5978a6355dea6070de1cd60cc2077e4ccca7ddbe Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:10:54 +0000 Subject: [PATCH 06/11] =?UTF-8?q?#29=20docs:=20CR=202=20=E2=80=94=20spec?= =?UTF-8?q?=20=C2=A74's=20refusal=20list=20names=20xpath=20on=20PDF=20and?= =?UTF-8?q?=20CSV?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/specs/2026-09-29-processor-service-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 3693668..1224164 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -307,7 +307,7 @@ Observo adopted in #395/#407) so the later plane can ingest them unchanged. | Malformed digest, or `input_uri` the input store does not recognize | `invalid_input`, terminal (issuer does not re-fetch) | yes | | Input bytes gone, or the download fails its checksum (`DataCorruption`, amended 2026-09-30) | `input_unreadable`, terminal for the command (issuer re-fetches, capped) | yes | | Bytes hash ≠ `input_digest` | `input_digest_mismatch`, terminal | yes | -| Extractor raises, including `MemoryError` under `RLIMIT_AS`, and a spec co-core refuses (`UnsupportedExtractionError` since 0.32.0, cannobserv#523: `regex`, `jsonpath`, an empty or invalid `xpath`, an unknown algorithm, `xpath` on HTML, `css` on XML / PDF / CSV), amended 2026-10-12 | `extraction_error`, terminal | yes | +| Extractor raises, including `MemoryError` under `RLIMIT_AS`, and a spec co-core refuses (`UnsupportedExtractionError` since 0.32.0, cannobserv#523: `regex`, `jsonpath`, an empty or invalid `xpath`, an unknown algorithm, `xpath` on HTML / PDF / CSV, `css` on XML / PDF / CSV), amended 2026-10-12 | `extraction_error`, terminal | yes | | Child timeout or crash | — | no; the 3rd attempt publishes `extraction_error`, terminal, and acks | | A non-transient exception escaping the handler (a bug, or an ack or dead-letter refused), amended 2026-09-30, 2026-10-02 and 2026-10-05 | at the cap, `extraction_error`, terminal, unless this entry's fact already went out (a fact published before a refused ack stands) | no; counted like a strike, and the 3rd attempt publishes, then dead-letters the entry with the exception as its reason. A dead-letter that does not land is retried alone, without running the command again (#28) | | GCS 5xx / 429 / timeout / auth; broker `NOPERM`; broker `OOM command not allowed`; broker `MISCONF` / `BUSY` / `MASTERDOWN` / `TRYAGAIN` / `CLUSTERDOWN` / `NOREPLICAS` (amended 2026-09-30) | — | no; the entry is reclaimed | From b666190e455722331d5cf9e9bc4054d4d8bd9548 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:11:04 +0000 Subject: [PATCH 07/11] =?UTF-8?q?#29=20test:=20CR=203=20=E2=80=94=20the=20?= =?UTF-8?q?wheelhouse=20builder's=20default=20tag=20is=20held=20to=20the?= =?UTF-8?q?=20pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 2 +- tests/test_pin.py | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 27d7810..67f7969 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -455,7 +455,7 @@ On boot, the service preflights both buckets and exits non-zero if either is unr co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-10-10, #47). Until watcher#350 it was in lockstep with Watcher. A bump changes `processor_version` on every fact, and Watcher re-baselines on that: -1. Rebuild the wheelhouse at the new tag, then update the pin in `pyproject.toml` and `EXPECTED` in `tests/test_pin.py`, and `uv lock`. +1. Rebuild the wheelhouse at the new tag, then update the pin in `pyproject.toml`, `EXPECTED` in `tests/test_pin.py` and the default tag in `scripts/build_wheelhouse.sh` (the pin test holds all three), and `uv lock`. 2. A release that adds an essence or an extraction rule gets its cases in `tests/fixtures/parity/cases.json` first, so their goldens come from co-core. Then regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved, and every case that started or stopped being refused. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. A catch-up across releases reads each CHANGELOG's Downstream lines for the extract, bus and blob-store surfaces, and diffs those packages' code between the tags (#29 did 0.19.7 → 0.32.0). 3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved. 4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models, `canonical_text` semantics and `resolve_dispatch_essence`. diff --git a/tests/test_pin.py b/tests/test_pin.py index 850b486..f322e97 100644 --- a/tests/test_pin.py +++ b/tests/test_pin.py @@ -5,6 +5,7 @@ with the goldens regenerated co-core-direct and every moved digest in the bump note. """ +import re import tomllib from importlib.metadata import version from pathlib import Path @@ -40,6 +41,14 @@ def test_installed_matches_the_pin(name: str) -> None: assert version(name) == EXPECTED +def test_the_wheelhouse_builder_defaults_to_the_pin() -> None: + # A fresh checkout runs it with no tag (AGENTS.md); a stale default builds wheels + # the lock can't resolve. + script = (ROOT / "scripts" / "build_wheelhouse.sh").read_text() + (default,) = re.findall(r'^tag="\$\{1:-(v[^}]+)\}"$', script, re.MULTILINE) + assert default == f"v{EXPECTED}" + + def test_processor_version_is_co_core_plus_generation() -> None: assert LOCAL_GENERATION == 1 # Watcher's LOCAL_EXTRACTION_GENERATION at the cutover assert PROCESSOR_VERSION == f"{EXPECTED}+1" From 4ea8950501c17d25c8b4c5182d1f8e01faf28fd7 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:11:20 +0000 Subject: [PATCH 08/11] =?UTF-8?q?#29=20test:=20CR=204=20=E2=80=94=20a=20ca?= =?UTF-8?q?se=20moving=20between=20digest=20and=20refused=20is=20in=20the?= =?UTF-8?q?=20bump=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- tests/test_gen_parity_goldens.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tests/test_gen_parity_goldens.py b/tests/test_gen_parity_goldens.py index fb6e1b8..89a16f9 100644 --- a/tests/test_gen_parity_goldens.py +++ b/tests/test_gen_parity_goldens.py @@ -126,6 +126,24 @@ def test_a_spec_co_core_refuses_is_recorded_as_refused(corpus: Path) -> None: assert doc["goldens"]["regex"] == {"refused": "UnsupportedExtractionError"} +def test_a_case_that_starts_or_stops_being_refused_is_listed(corpus: Path) -> None: + # The bump note's view of e.g. cannobserv#557, when HTML starts honouring xpath. + doc = json.loads((corpus / "goldens.json").read_text()) + was_refused = doc["goldens"]["refused_xpath_on_html"] + was_extracted = doc["goldens"]["xml_xpath"] + doc["goldens"]["refused_xpath_on_html"] = was_extracted + doc["goldens"]["xml_xpath"] = was_refused + (corpus / "goldens.json").write_text(json.dumps(doc)) + + moved = gen.main(corpus) + + assert "refused_xpath_on_html refused: None -> UnsupportedExtractionError" in moved + assert "xml_xpath refused: UnsupportedExtractionError -> None" in moved + assert ( + f"xml_xpath content_fingerprint: None -> {was_extracted['content_fingerprint']}" + ) in moved + + def test_a_malformed_xml_body_is_not_a_refusal(corpus: Path) -> None: # A parse failure is a plain ValueError: it names its case, never becomes a golden. spec = {"extraction": {"algorithm": "full_page"}} From efbabb9bb78978e24f5d10146fc7424d750e2aec Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:11:28 +0000 Subject: [PATCH 09/11] =?UTF-8?q?#29=20test:=20CR=205=20=E2=80=94=20the=20?= =?UTF-8?q?corpus=20names=20each=20layer's=20refusals,=20not=20a=20count?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- tests/processors/test_extract.py | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tests/processors/test_extract.py b/tests/processors/test_extract.py index c33e635..dd314d6 100644 --- a/tests/processors/test_extract.py +++ b/tests/processors/test_extract.py @@ -98,7 +98,22 @@ def test_the_corpus_covers_the_refusals_and_the_xml_essences() -> None: assert {"text/xml", "application/xml", "application/rss+xml", "application/atom+xml"} <= ( essences ) - assert len(REFUSED) >= 10 + # extraction_config_from_spec refuses these whatever the essence... + assert { + "refused_regex", + "refused_jsonpath", + "refused_empty_xpath", + "refused_unknown_algorithm", + } <= REFUSED + # ...and the dispatched extractor these. xpath on HTML moves at cannobserv#557. + assert { + "refused_xpath_on_html", + "refused_xpath_on_csv", + "refused_css_on_xml", + "refused_css_on_pdf", + "refused_css_on_csv", + "refused_invalid_xpath", + } <= REFUSED def test_the_feed_golden_is_co_cores_own_pin() -> None: From 9e36d955805ad18430fbb44464396a98adb50e90 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:11:32 +0000 Subject: [PATCH 10/11] =?UTF-8?q?#29=20docs:=20CR=206=20=E2=80=94=20the=20?= =?UTF-8?q?amendment=20says=20the=20Watcher=20coordination=20is=20in=20pro?= =?UTF-8?q?gress?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/specs/2026-09-29-processor-service-design.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 1224164..9f1200c 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -58,8 +58,9 @@ on `co-processor`. cannobserv#523 (Sections 2 and 5). The four XML essences dispatch to co-core's `XmlExtractor`. A spec the dispatched extractor can't honour raises `UnsupportedExtractionError`, a terminal `extraction_error` (Section 4). -- The bump changes `resolve_dispatch_essence`, so it was coordinated with Watcher - (watcher#366, Section 5). +- The bump changes `resolve_dispatch_essence`, so it is coordinated with Watcher + on watcher#366: Processor posted on 2026-10-12, and Watcher's acknowledgement + gates the deploy (Section 5). - The corpus covers the XML essences, and a refusal is a golden (Section 7). --- From 3b6f785d4755f850a82b17fd2c56b8098bd197ad Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Mon, 12 Oct 2026 18:21:07 +0000 Subject: [PATCH 11/11] =?UTF-8?q?#29=20test:=20CR=209=20=E2=80=94=20the=20?= =?UTF-8?q?pin=20test's=20docstring=20names=20the=20wheelhouse=20default?= =?UTF-8?q?=20tag?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- tests/test_pin.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/test_pin.py b/tests/test_pin.py index f322e97..404c624 100644 --- a/tests/test_pin.py +++ b/tests/test_pin.py @@ -1,8 +1,9 @@ """The co-core pin is exact, and Processor owns it (spec D5, amended 2026-10-10; §5). A bump changes ``processor_version`` on every fact, and Watcher re-baselines on it, so -it must be a deliberate act: edit ``EXPECTED`` here, in the same commit as the pin, -with the goldens regenerated co-core-direct and every moved digest in the bump note. +it must be a deliberate act: edit ``EXPECTED`` here, in the same commit as the pin +and ``scripts/build_wheelhouse.sh``'s default tag, with the goldens regenerated +co-core-direct and every moved digest in the bump note. """ import re