Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29

Activity

1. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Re-entry test restated (#47, 2026-10-10). CannObserv/watcher#350 deleted Watcher's extraction, and D5 now reads: Processor owns the co-core pin (spec D5 and §5, as amended in #49). That supersedes "in step / in lockstep with Watcher" here.

   Re-entry: a cannobserv release that contains CannObserv/cannobserv#523. That alone; Watcher being ready to take it is no longer a condition.

   At the bump:

   1. Pin, EXPECTED in tests/test_pin.py, uv lock. Ask 2 stands: don't hand-bump LOCAL_GENERATION.
   2. Add the feed fixture (e.g. Tailscale's security bulletins) to tests/fixtures/parity/cases.json first, so its golden comes from co-core and not from Processor.
   3. Regenerate the goldens co-core-direct: uv run python scripts/gen_parity_goldens.py tests/fixtures/parity. It prints every moved digest. List them all in the bump note as deliberate re-baselines.
   4. The real corpus (real/) passes unchanged, or the bump note says which recorded fingerprints moved. The check for #523's xpath / regex / jsonpath refusal stays.
   5. Notice to Watcher, because processor_version moves. If the release also changes the contract (the content.process / content.derived models, canonical_text semantics, or resolve_dispatch_essence), coordinate with Watcher first instead of just giving notice.

   Procedure: docs/DEPLOYMENT.md § Bumping co-core.
2. gregoryfoster

   added a commit that references this issue on Oct 10, 2026

   #47 fix: CR 4 — a case whose extractor raises is named, and nothing i…

   ...

   d4bae35
3. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Catch-up folded in (operator, 2026-10-10T23:36Z, at epic #24's check-in). Since #47, the co-core pin is Processor's own. co-core is at v0.30.0 against the 0.19.7 pin. There's no separate catch-up issue: whichever of #29 or #37 bumps first does the full jump. That means reading every CHANGELOG from 0.20 to the target release (the Downstream lines for the extract, bus and blobstore surfaces, plus security fixes), and listing anything that changes the content.process / content.derived contract. A contract change needs coordination with Watcher first, because Watcher still pins 0.19.7 for contracts (D5 as amended). The other issue then bumps from wherever the first left the pin.
4. gregoryfoster commented on Oct 12, 2026

   

   gregoryfoster

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Re-entry test met: cannobserv v0.32.0 (tag v0.32.0, 33e781a6) contains CannObserv/cannobserv#523.

   This is your step 5's coordinate-first case. The release changes resolve_dispatch_essence: .xml / .rss / .atom URLs behind a missing or ambiguous header now resolve to the XML essences. Watcher resolves the essence for the command, and Processor never re-resolves it. So until Watcher takes the same release, such an item still reaches Processor as the header's essence and extracts as HTML. Filed as CannObserv/watcher#366. A feed with an informative header dispatches correctly from your side alone, and that includes the Tailscale fixture's text/xml.

   What moves for Processor, checked against main (c50b343):

   • Dispatch: text/xml, application/xml, application/rss+xml and application/atom+xml go to co_core.pure.extract.xml_feed.XmlExtractor, where they took the HTML fallback.
     • Under full_page, an RSS or Atom root gives one id<TAB>date<TAB>title<TAB>link chunk per item, with no channel fields. A feed with no items is empty.
     • Any other root gives its text. An xpath spec gives one chunk per match.
   • Refusals: UnsupportedExtractionError(ValueError), raised by extraction_config_from_spec for regex, jsonpath, an empty xpath or an unknown algorithm, and by the extractor for xpath on HTML or css on XML / PDF / CSV. processors/extract.py calls both with no catch, so your child runner reports a refused spec as extraction_error. The token's definition in changes.py now names this case; the Literal is unchanged. Exposure measured 2026-10-05: 4 prod items, all text/html + css, so no live spec moves.
   • No call-site change. scripts/gen_parity_goldens.py runs the same translate, merge, dispatch flow. Ask 2 stands: don't hand-bump LOCAL_GENERATION.

   For the parity run:

   • The trimmed Tailscale feed co-core's goldens use is at tests/co_core/pure/extract/fixtures/tailscale_security_bulletins.xml in cannobserv. Its canonical text is pinned at sha256:78e284978faf41fbe8ccd8dc06e6d2803d9788097fca633486dfbbbdc024be09.
   • cannobserv's tests/co_core/pure/extract/test_dispatch.py now runs your exact chain ({**extraction_config_from_spec(spec), **extraction_overrides_for_essence(essence)}, then extractor_for_essence(essence).extract) over refused and honoured pairs, a useful checklist for the refusal check.

   The full jump is 0.19.7 → 0.32.0, per the operator's catch-up note. The [0.32.0] CHANGELOG entry's Downstream lines are the #523 part of that reading.
5. gregoryfoster commented on Oct 12, 2026

   

   gregoryfoster

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Re-entry met: cannobserv v0.32.0 contains CannObserv/cannobserv#523 (released 2026-10-12)

   This is the coordinate-first kind, not a notice.

   • v0.32.0 changes resolve_dispatch_essence: .xml → application/xml, .rss → application/rss+xml and .atom → application/atom+xml join the extension tiebreak.
   • Watcher computes the essence that goes on every content.process command, with its own co-core (==0.19.7, through src/core/media_type.py).
   • So until Watcher takes the same release, a feed URL with a missing or ambiguous Content-Type arrives with the old essence and is extracted as HTML. Tailscale's feed sends text/xml, so it dispatches correctly either way.
   • The Watcher side needs a new Watcher issue: watcher#342 is closed, and its 2026-10-10 correction says this bump arrives as Processor's coordination ask.

   What the catch-up jump crosses (0.19.7 → 0.32.0, checked against the code, not only the CHANGELOG):

   • content.process / content.derived models: unchanged. changes.py's only change is a docstring; extraction_error now also covers a refused spec.
   • canonical_text: unchanged.
   • The blob store and the bus I/O: unchanged.
   • The extract package:
     • the new XmlExtractor and four XML essences;
     • the new UnsupportedExtractionError(ValueError). extraction_config_from_spec refuses regex, jsonpath, an empty xpath and an unknown algorithm. HtmlExtractor refuses xpath (until cannobserv#557), and XML, PDF and CSV refuse a non-empty selectors.
   • extract.py calls these uncaught, so a refused spec reports extraction_error with no code change.

   At the bump (your 2026-10-10 checklist still stands):

   • Don't hand-bump LOCAL_GENERATION.
   • Add the feed fixture before regenerating the goldens. A live check for reference: the Tailscale feed fetched 2026-10-12 gives 46 chunks, and two fetches about 2.5 hours apart give the same fingerprint, sha256:d387e02e…. The fingerprint will move as new bulletins appear, so take it from your own fixture copy.
   • The real/ corpus should pass unchanged: the 4 prod items are text/html + css, and no XML or PDF/CSV selector specs were measured on 2026-10-05.
   • The parity run checks the refusals.

   Upstream detail: CannObserv/cannobserv#523 (decisions, ship summary and verification) and CHANGELOG [0.32.0].
6. gregoryfoster

   added 8 commits that reference this issue on Oct 12, 2026

   #366 chore: take co-core 0.32.0 in step with Processor

   ...

   b6bfd38

   #29 docs: CR 1 — spec §5's floor sentence goes back on its parent bullet

   ...

   9210b92

   #29 docs: CR 2 — spec §4's refusal list names xpath on PDF and CSV

   ...

   5978a63

   #29 test: CR 3 — the wheelhouse builder's default tag is held to the pin

   ...

   b666190

   #29 test: CR 4 — a case moving between digest and refused is in the b…

   ...

   4ea8950

   #29 test: CR 5 — the corpus names each layer's refusals, not a count

   ...

   efbabb9

   #29 docs: CR 6 — the amendment says the Watcher coordination is in pr…

   ...

   9e36d95

   #29 test: CR 9 — the pin test's docstring names the wheelhouse defaul…

   ...

   3b6f785
7. gregoryfoster commented on Oct 12, 2026

   

   gregoryfoster

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Shipped in #51 (415fa24..3b6f785, fast-forwarded to main), deployed 2026-10-12T18:33Z, processor_version 0.32.0+1.

   • Pin: co-core / co-core-aio / co-core-sync went from ==0.19.7 to ==0.32.0, the catch-up jump. LOCAL_GENERATION stays 1 (ask 2). The lock moved only those three packages.
   • Catch-up reading (git diff v0.19.7 v0.32.0 plus the CHANGELOG):
     • The bus, the stores, canonical_text, spec_fingerprint and the content.process / content.derived models are unchanged in code.
     • No security fixes in the range.
     • The one contract change is resolve_dispatch_essence, coordinated on watcher#366. Watcher acknowledged and was already on 0.32.0.
   • Parity (ask 3):
     • The feed fixture (co-core's trimmed Tailscale feed) went in before the goldens were regenerated directly from co-core. Its golden equals co-core's hand-pinned sha256:78e28497….
     • Also added: an Atom feed, a feed with no items, a generic XML document and an xpath on XML.
     • A spec co-core refuses is now a golden, {"refused": "UnsupportedExtractionError"}, with 10 refusal cases covering both layers. Through the child, each comes back raised, which the handler publishes as extraction_error, terminal.
     • Bump note: 16 cases added; none of the 13 existing goldens moved. The real corpus passes unchanged, 9/9.
   • Live check, passed: the first command from each source after the deploy completed.
     • The 4 existing sources produced the same output as on 0.19.7: 1e5e92de…, fbae5c98…, cd8d7281…, 36e8c9f2…. No spec was refused.
     • The new Tailscale source (…TEC6WZ5, 19:27:02Z) gave sha256:d387e02e…, 6485 bytes, 46 lines. That equals the live-fetch fingerprint in the comment above, and my own extraction of the feed fetched at 17:3xZ.
   • Review: CR 1–9 are on main; the PR body lists them.

   Follow-ups: none from here. cannobserv#557 (HTML honouring xpath) will move refused_xpath_on_html at its bump. The tests and the bump procedure now expect that.
8. gregoryfoster

   closed this as completedon Oct 12, 2026
9. gregoryfoster commented on Oct 12, 2026

   

   gregoryfoster

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Post-ship: first new source on 0.32.0+1 (archiver#317). The Tailscale security-bulletin feed (InfoSource 01M4RS9CWFZCDNRJXTPTEC6WZ5, https://tailscale.com/security-bulletins/index.xml, text/xml), registered 2026-10-12T19:26:31Z, an hour after this deploy.

   From processor's journald, one ack: complete record, the only one for this source:

   field	value
   timestamp	2026-10-12T19:27:02.868250Z, attempt 1
   command_id	01M4RSAAEEKYWF1BYHFK90ETNY
   input_digest	05227afaf92920de7acf1ec6919ab428adb1c69050f84003bff54fab2afdb135
   output_digest	sha256:d387e02ee8c46a038ec6c1ea8c9b63b3954827d939e86a24a385fc51ff53f50d (6485 bytes, empty: false)
   processor_version	0.32.0+1 (build 3b6f785d4755)
   timings (ms)	read 63.4 · extract 530.0 · store 105.8 · publish 1.9 · total 701.7

   • Outcome: content.derived, not processing_failed. Store → publish → ack, so the XADD returned before the ack was logged. Nothing is pending, retried or dead-lettered.
   • Format: one line per feed item (0.32.0). Archiver's own 0.32.0 extraction of the URL gives the same digest and size. This source was never derived on 0.19.7, so the bump produces no format-change revision for it.
   • Essence: Processor dispatches on the command's media_type as Watcher issued it and does not log it; Watcher holds that value.
   • Missing content.revisions: the gap is downstream of Processor, in Watcher's handling of content.derived for this command. Watcher's info.watch-status came ~22 ms after Processor's ack. Followed up on archiver#317.