Skip to content

JSON: take co-core's JsonExtractor (cannobserv#354) and add a jsonpath parity fixture #37

Closed

Description

@gregoryfoster

Moved from CannObserv/watcher#212, which closes as superseded. JSON extraction is no longer Watcher's concern.

The problem (verified on co-core 0.10.0, still true on 0.19.7's table)

application/json isn't in co-core's EXTRACTOR_BY_ESSENCE, so it falls back to HtmlExtractor. That extractor ignores a jsonpath spec and fingerprints the whole document:

raw = b'{"items": [{"title": "Rule 5", "status": "active"}]}'
HtmlExtractor().extract(raw, {"algorithm": "jsonpath", "selector": "$.items[*].title"})
# -> one chunk, text = the entire document. Selector ignored.

The chunks aren't empty, so the empty-extraction guard never trips. Any unrelated field, such as a timestamp or a counter, then reads as a change. Archiver's schema allows {json: jsonpath}, so a JSON source can legitimately carry such a spec.

Why it lands here

Ask (when CannObserv/cannobserv#354 releases)

Exposure today: none. All 4 production items are text/html, measured 2026-10-05 on CannObserv/watcher#342.

Refs: CannObserv/cannobserv#354, CannObserv/watcher#212, CannObserv/watcher#342.

Activity

  1. gregoryfoster commented on Oct 10, 2026

    @gregoryfoster
    ContributorAuthor

    Re-entry test restated (#47, 2026-10-10). CannObserv/watcher#350 deleted Watcher's extraction, and D5 now reads: Processor owns the co-core pin (spec D5 and §5, as amended in #49). That supersedes "in lockstep with Watcher" here (and its "as with #29").

    Re-entry: a cannobserv release that contains CannObserv/cannobserv#354. That alone; Watcher being ready to take it is no longer a condition.

    At the bump:

    1. Pin, EXPECTED in tests/test_pin.py, uv lock. Ask 2 stands: don't hand-bump LOCAL_GENERATION.
    2. Add the jsonpath fixture (and the +json vendor-type one, if co-core maps that family) to tests/fixtures/parity/cases.json first, so its golden comes from co-core and not from Processor.
    3. Regenerate the goldens co-core-direct: uv run python scripts/gen_parity_goldens.py tests/fixtures/parity. It prints every moved digest. List them all in the bump note as deliberate re-baselines.
    4. The real corpus (real/) passes unchanged, or the bump note says which recorded fingerprints moved. The fixture still asserts that jsonpath narrows the canonical text to the selected nodes. It may be the same bump as Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29.
    5. Notice to Watcher, because processor_version moves. If the release also changes the contract (the content.process / content.derived models, canonical_text semantics, or resolve_dispatch_essence), coordinate with Watcher first instead of just giving notice.
    6. Expect one synthetic golden to move: unknown_essence_is_html uses application/json as its "unknown" essence. Once JSON dispatches to JsonExtractor, that case stops testing the HTML fallback. Move it to an essence that is still unknown, and list the moved digest.

    Procedure: docs/DEPLOYMENT.md § Bumping co-core.

  2. gregoryfoster commented on Oct 10, 2026

    @gregoryfoster
    ContributorAuthor

    Catch-up folded in (operator, 2026-10-10T23:36Z, at epic #24's check-in). Since #47, the co-core pin is Processor's own. co-core is at v0.30.0 against the 0.19.7 pin. There's no separate catch-up issue: whichever of #29 or #37 bumps first does the full jump. That means reading every CHANGELOG from 0.20 to the target release (the Downstream lines for the extract, bus and blobstore surfaces, plus security fixes), and listing anything that changes the content.process / content.derived contract. A contract change needs coordination with Watcher first, because Watcher still pins 0.19.7 for contracts (D5 as amended). The other issue then bumps from wherever the first left the pin.

  3. gregoryfoster commented on Oct 12, 2026

    @gregoryfoster
    ContributorAuthor

    From the burn-down epic #24, 2026-10-12. The co-core catch-up is done: #29 (#51) took co-core 0.19.7 → 0.32.0, read the whole range, and deployed as processor_version 0.32.0+1. This issue's bump now starts from 0.32.0, so it reads only the CHANGELOGs after that. Ignore the earlier note saying this bump might carry the catch-up.

    #29 also set the pattern to follow:

    Its trigger was not met by v0.32.0: the upstream issue is still open.

  4. changed the title [-]JSON: take co-core's JsonExtractor (cannobserv#354) in lockstep with Watcher, and add a jsonpath parity fixture[/-] [+]JSON: take co-core's JsonExtractor (cannobserv#354) and add a jsonpath parity fixture[/+] on Oct 20, 2026
  5. gregoryfoster commented on Oct 20, 2026

    @gregoryfoster
    ContributorAuthor

    cannobserv v0.38.0 is released: this issue's re-entry trigger is met

    v0.38.0 (MINOR) contains CannObserv/cannobserv#354, along with CannObserv/cannobserv#557 and CannObserv/cannobserv#570 (release vehicle CannObserv/cannobserv#582). Publish is green on the tag (run 39353767748), and all 4 wheels and 4 sdists are in the wheelhouse.

    Changelog: https://github.com/CannObserv/cannobserv/blob/v0.38.0/CHANGELOG.md#0380---2026-10-19

    The range from your pin. pyproject.toml pins ==0.37.0, so the bump reads two entries: 0.37.1 (bus-only: the capture contracts and the info_item_primary_changed key) and 0.38.0.

    Title corrected. I've edited this issue's title to drop "in lockstep with Watcher". Since #47, D5 makes the co-core pin Processor's own. Watcher's part of CannObserv/cannobserv#354 is the .json tiebreak in resolve_dispatch_essence, and that is notice, not lockstep (see below).

    Goldens that move

    I replayed your cases.json inputs through v0.38.0's translator and extractor_for_essence, merged in the same order as processor.processors.extract (Processor main at 9244fd0). Three cases change, one more than the earlier notes here list:

    1. refused_jsonpath now extracts. This one isn't in the earlier notes. CannObserv/cannobserv#354's translator maps jsonpath to {"jsonpath": selector}, where it used to refuse the algorithm. On items.json / application/json, $.items now gives one match chunk labelled $['items'], holding the array's canonical JSON: [{"id":1,"title":"Agenda item one"},{"id":2,"title":"Item two"}].
      • test_the_corpus_covers_the_refusals_and_the_xml_essences asserts that it's in REFUSED, under "extraction_config_from_spec refuses these whatever the essence", so the test fails on the bump. Take it out of that set. Since it isn't a refusal any more, renaming the case is your call.
      • For Ask 2 ("narrows the canonical text to the selected nodes"): $.items selects the whole array, so it doesn't narrow much. $.items[*].title on the same input gives two chunks, Agenda item one and Item two, labelled $['items'][0]['title'] and $['items'][1]['title'].
      • To keep a jsonpath refusal golden, put the spec on a non-JSON body: jsonpath on text/html (agenda.html) is refused, now by HtmlExtractor rather than the translator. A css spec on items.json is refused by JsonExtractor too.
    2. refused_xpath_on_html now extracts (CannObserv/cannobserv#557). //p on agenda.html gives two chunks identical to the css spec p: Section 1 and Section 2. Move it out of the dispatched-extractor refused set in the same test; its comment already says "xpath on HTML moves at cannobserv#557".
    3. unknown_essence_is_html no longer tests the HTML fallback (the earlier note's item 6). application/json now dispatches to JsonExtractor, which gives one document chunk labelled $ with the canonical JSON. An essence that isn't in EXTRACTOR_BY_ESSENCE (for example application/x-unknown) still falls back to HtmlExtractor on the same bytes.

    scripts/gen_parity_goldens.py will print every moved digest. Per the #29 pattern, list them all in the bump note. Nothing else should move: no other case uses a JSON essence, jsonpath or ignore_selectors.

    CannObserv/cannobserv#570

    No change is needed. An uncompilable CSS selector now raises UnsupportedExtractionError, where it raised soupsieve's SelectorSyntaxError or NotImplementedError. Your child already reported those as extraction_error. A parity case for ignore_selectors (HTML only; XML, PDF and CSV refuse a non-empty list) is a suggestion, not an ask.

    Order

    Refs CannObserv/cannobserv#582, CannObserv/cannobserv#354, CannObserv/cannobserv#557, CannObserv/cannobserv#570.

  6. gregoryfoster commented on Oct 20, 2026

    @gregoryfoster
    ContributorAuthor

    Hand-off from the burn-down epic #24, 2026-10-20. This is next: the trigger fired. cannobserv v0.38.0 (tag 33676e6, 2026-10-19) contains CannObserv/cannobserv#354 (closed 23:39Z), plus CannObserv/cannobserv#557 and CannObserv/cannobserv#570. Read the release comment above first: its three moved goldens hold against cases.json and tests/processors/test_extract.py on origin/main (9244fd0). This brief adds what the coordinator checked, and one trap the comment misses.

    What changed from the body's plan:

    • Not lockstep. The title is already corrected; the body's first box still says "in lockstep with Watcher". D5 as amended by Close out the cutover: spec §6 step 5 done, and re-decide D5 (lockstep pin, Watcher-generated goldens) now that Watcher doesn't extract #47 applies: Processor owns the pin, Watcher gets notice.

    • The resolve_dispatch_essence change is notice, not coordination. D5 names that function as a reason to coordinate. But Processor never calls it (grep -rn resolve_dispatch_essence src/: no hits): Watcher resolves the essence on its own pin, and Processor dispatches whatever arrives. Either order is safe:

      • an old Watcher keeps sending the ambiguous essence, which goes to HTML as before;
      • a new Watcher's application/json reaches HTML on today's Processor, and JsonExtractor after this bump.

      State that reasoning in the bump note. CannObserv/cannobserv#354's side has already posted its notice on [Epic] Headless Watcher and cohort boundaries watcher#368.

    • Ask 3 (+json): co-core maps two named types, application/ld+json and application/vnd.api+json, not the suffix. An unlisted +json type still goes to HTML, where a jsonpath spec is refused.

    The bump: 0.37.0 → 0.38.0 (pyproject.toml:11-13, all three packages), so processor_version becomes 0.38.0+1. LOCAL_GENERATION stays 1. Follow #21's pattern and DEPLOYMENT.md's "Bumping co-core".

    • Pins held by tests: tests/test_pin.py:18 and scripts/build_wheelhouse.sh's default tag.
    • Catch-up reading, read in v0.38.0's CHANGELOG by the coordinator:
      • 0.37.1 is bus-only: the content.capture contracts on content.artifacts, and info_item_primary_changed's key. Processor reads neither stream. Confirm the content.process / content.derived models are unchanged, in code.
      • 0.38.0's wire is unchanged (CHANGELOG: canonical_text, both models, spec1:).
    • New dependency: python-jsonpath>=2.2,<3 (from PyPI, no dependencies of its own) enters uv.lock and runs in the contained child. tests/test_child.py's whole-corpus test reads cases.json, so the JSON cases exercise its import under containment. Confirm the pytest header says required.
    • JSON parse failures raise ValueError("Failed to parse JSON: …"), the PDF/XML shape. This covers NaN, Infinity, 1e400, too-deep nesting, a .. over 100 levels, and an unpaired surrogate. Confirm they take the same terminal path as a PDF parse failure (processors/extract.py:68, handler.py:187/256).

    Goldens (co-core-direct, scripts/gen_parity_goldens.py): expect exactly three to move. List every digest the script prints in the bump note.

    1. refused_jsonpath now extracts ($.items: one match chunk, $['items']). Take it out of the translator-refusal set in test_the_corpus_covers_the_refusals_and_the_xml_essences. Renaming it is your call.
    2. refused_xpath_on_html now extracts (two chunks, identical to the css spec p). Take it out of the dispatched-extractor set. Its comment already says it moves at #557.
    3. unknown_essence_is_html now reaches JsonExtractor (one document chunk, $). The fallback it was named for needs a new case, for example application/x-unknown on the same bytes.

    Real corpus: all 9 items are text/html (export.json), so expect 0 moved. Never regenerate it.

    New cases (Asks 2 and 3, and refusal coverage the bump would otherwise lose):

    • jsonpath $.items[*].title on items.json: two chunks, Agenda item one and Item two. This is the "narrows" assertion.
    • A +json vendor type (application/ld+json or application/vnd.api+json).
    • jsonpath on agenda.html, refused by HtmlExtractor, and css on items.json, refused by JsonExtractor. These keep a jsonpath refusal golden in the corpus.
    • Optional (CannObserv/cannobserv#570): an ignore_selectors case on HTML.

    Traps:

    1. The comment misses one: tests/test_gen_parity_goldens.py:129 (test_a_case_that_starts_or_stops_being_refused_is_listed). It swaps refused_xpath_on_html's golden with xml_xpath's, and asserts that refused_xpath_on_html goes from extracted back to refused. On 0.38.0 both cases extract, so the assertions fail. Re-point it at a case that stays refused, such as refused_css_on_xml, and update its comment.
    2. The live gate is Dead-letter give-ups: adopt co-core 0.37.0's dead_lettered reason (cannobserv#536), with the 0.32.0 → 0.37.0 bump #21's: each source's first 0.38.0+1 digest should equal its last 0.37.0+1 digest. All production items are text/html; #557 is additive, and #570 only changes how an uncompilable selector fails. So any moved digest is a finding: stop and report it.
    3. Interaction with Follow the broker to Valkey 7.2 (CannObserv/broker#85): CI image, scratch server, 7.0.15-only claims #54: none. If the broker's window lands first, finish whichever deploy is in flight before starting the other.

    Gates (operator): each cross-repo post needs approval.

    The deploy itself goes through #34's gate.

    Done when:

    • the pin is ==0.38.0 on all three packages, and processor_version is 0.38.0+1;
    • goldens are regenerated, with the three moves above listed by digest. If anything else moves, it is explained or the work stops;
    • the new cases above are in the corpus, including the narrowing assertion, and trap 1's test is re-pointed;
    • the bump note gives:
      • the catch-up reading, checked in code;
      • the moved goldens;
      • the real corpus (0 moved);
      • the either-order reasoning;
    • the spec (D5's "today" pin, §5 and the amendment block), AGENTS.md and DEPLOYMENT.md say 0.38.0;
    • the Watcher notice is posted, with approval, before the deploy;
    • the deploy is checked live (trap 2);
    • the Archiver notice is posted, with approval, after the live checks.

    Out of scope:

    Ship: issue → branch in a worktree → plan (short) → TDD → PR (Refs #37: the notices and the deploy come after the merge) → FF merge → git pull --ff-only → the Watcher notice → scripts/deploy.sh → the live checks → the Archiver notice. The closing comment gives each box with its evidence, the known limits, and the follow-ups (or "none").

    Kickoff prompt:

    Pick up processor#37. Read its hand-off comment from epic #24 first.

  7. gregoryfoster commented on Oct 20, 2026

    @gregoryfoster
    ContributorAuthor

    From the burn-down epic #24, 2026-10-20: the operator has approved both cross-repo posts in the hand-off above. Post them without asking again. The order and conditions still apply:

    The approval covers these two posts only. Any other cross-repo post still needs its own approval. Link each post in the closing comment.

  8. gregoryfoster commented on Oct 20, 2026

    @gregoryfoster
    ContributorAuthor

    Bump note: co-core 0.37.0 → 0.38.0, processor_version 0.38.0+1. PR #56, fast-forward merged as 540975e. Deployed at 2026-10-20T04:29:44Z.

    Catch-up reading, checked in code (git diff v0.37.0 v0.38.0 -- packages/; 0.37.1 between):

    • The content.process / content.derived models (ContentProcessCommand(Emit), ProcessingComplete*, ProcessingFailed*, ProcessingFailureReason): identical once docstrings are stripped (AST compare). The rest of changes.py is the capture pair (cannobserv#571).
    • Bus: envelope.py registers the capture pair and moves info_item_primary_changed's key to …:occurred_at (#578). streams.py adds CONTENT_CAPTURE. Processor reads neither. dead_letter, exceptions, co_core_aio and co_core_sync change only their version strings.
    • canonical_text, the blob store, hashing: unchanged.
    • spec_fingerprint: its serialization moved into the shared _canonical_json, with the same json.dumps arguments. No spec1: golden moved.
    • Extract: JsonExtractor (#354), xpath on HTML (#557), ignore_selectors and the refusal of an uncompilable selector (#570), the dispatch map, the translator, and .json in resolve_dispatch_essence.
    • Dependencies: python-jsonpath>=2.2,<3 (locks 2.2.2) runs in the contained child. The whole-corpus child test runs every JSON case under containment required. soupsieve is now declared; it was already installed.
    • CHANGELOG 0.37.1 and 0.38.0: no security entries.

    Moved goldens. On main's corpus, unchanged, exactly three cases move:

    • refused_jsonpath: refused → sha256:d6be94c49f4d0c9c4084abadf66e4974c89e591c729aa792438ee51d47dee587 (64 B)
    • refused_xpath_on_html: refused → sha256:c331fdd4416460510d486e08aa51245cf42b53482beee1e70f2f369dff81962f (90 B)
    • unknown_essence_is_html: sha256:52492823a71e22338921f03375537ca3f356a747c98a72c30795b33c9d852d45 (82 B) → sha256:d1da45850448ef0dc317a67de8685e7a6e505db6fe94a1cd553786a888f74eac (74 B, now JsonExtractor)

    The committed regeneration printed co-core 0.38.0: 12 moved; every line is a case added or removed.

    • Renamed: refused_jsonpath → json_jsonpath and refused_xpath_on_html → html_xpath, with the digests above.
    • Re-pointed: unknown_essence_is_html now uses application/x-unknown. It takes the HTML fallback again and keeps its v1 digest 52492823…. A new test holds that essence outside EXTRACTOR_BY_ESSENCE.
    • Added (extract):
      • json_full_page and json_vnd_api: both d1da4585…
      • json_jsonpath_titles: 7ac46158…. Its text is exactly Agenda item one\nItem two; this is Ask 2's narrowing check.
      • html_ignore_selectors: 54fb39f5…
    • Added (refused): refused_empty_jsonpath, refused_jsonpath_on_html, refused_jsonpath_on_unlisted_plus_json (application/problem+json goes to HTML) and refused_css_on_json.

    Real corpus: 9/9 unchanged (all text/html).

    Either adoption order is safe. D5 names resolve_dispatch_essence as a reason to coordinate, but Processor never calls it (grep -rn resolve_dispatch_essence src/: no hits). Watcher resolves the essence on its own pin, and Processor dispatches whatever arrives:

    • An old Watcher keeps sending the ambiguous essence, which reaches HTML, as before.
    • A new Watcher's application/json reaches HTML on the old Processor, and JsonExtractor from this deploy on.

    So this bump is notice, not coordination. The notice was posted with approval before the deploy: CannObserv/watcher#368 (comment)

    JSON parse failures (NaN, Infinity, 1e400, too-deep nesting, an unpaired surrogate, a .. query past the depth limit) raise ValueError, not a refusal. Through the contained child they come back raised / ValueError:, a PDF parse failure's path, which the handler publishes as extraction_error, terminal.

    Trap 1: test_a_case_that_starts_or_stops_being_refused_is_listed now points at refused_css_on_xml.

    Live so far:

    • REVISION is 540975e10156. starting shows processor_version 0.38.0+1, child_containment required, Landlock ABI 6. NRestarts=0.
    • The digest gate is pending: each source's first 0.38.0+1 digest must equal its last 0.37.0+1 digest. Baseline:
    info_source_id Last 0.37.0+1 fact output_digest
    01KRYSPTN633JZ8582NHHSAH9A 2026-10-20T03:37:02Z sha256:80ea56dd…
    01KTCNXKXN7Z94G495YEGM6JX3 2026-10-19T19:34:03Z sha256:f77c3471…
    01KTVPAV7RW8TDZH88A43AFT04 2026-10-19T19:34:02Z sha256:8f0f09f6…
    01KVP9B9HVZ9NK1HRWT5569R62 2026-10-19T22:31:04Z sha256:36e8c9f2…
    01M4RS9CWFZCDNRJXTPTEC6WZ5 2026-10-19T22:49:02Z sha256:f5339c1d…

    The Archiver notice follows the live checks.

  9. gregoryfoster commented on Oct 20, 2026

    @gregoryfoster
    ContributorAuthor

    Closing: shipped, deployed, and checked live. PR #56 was fast-forward merged as 540975e and deployed at 2026-10-20T04:29:44Z. The bump note is above.

    Done when:

    • The pin is ==0.38.0 on all three packages, and processor_version is 0.38.0+1. pyproject.toml, uv.lock, tests/test_pin.py EXPECTED and scripts/build_wheelhouse.sh's default tag all moved. LOCAL_GENERATION stays 1. python-jsonpath 2.2.2 is in the lock.
    • The goldens are regenerated co-core-direct, with the three moves listed by digest. On the unchanged corpus exactly refused_jsonpath, refused_xpath_on_html and unknown_essence_is_html moved. The committed regeneration lists only cases added or removed by the renames. Nothing else moved.
    • The new cases are in the corpus, and trap 1's test is re-pointed.
      • Narrowing: json_jsonpath_titles gives exactly Agenda item one\nItem two.
      • The +json vendor type: json_vnd_api.
      • The jsonpath refusals: on HTML, and on an unlisted +json type.
      • css refused on JSON, and an empty jsonpath refused.
      • html_ignore_selectors.
      • The fallback case is held to an unmapped essence.
      • JSON parse failures and the .. depth limit raise ValueError, and come back raised through the contained child.
      • Trap 1's test now points at refused_css_on_xml.
    • The bump note gives the catch-up reading (checked in code), the moved goldens, the real corpus (9/9 unchanged) and the either-order reasoning.
    • Docs. The spec has a 2026-10-20 amendment, and D5, §2, §4's refusal row, §5, §7, the quick reference and OQ2 are updated. AGENTS.md's pin and Owns line are updated. DEPLOYMENT.md names no current pin, so it needed no change.
    • The Watcher notice was posted, with approval, before the deploy: [Epic] Headless Watcher and cohort boundaries watcher#368 (comment)
    • Deployed, and the live checks pass.
      • REVISION is 540975e10156. starting shows processor_version 0.38.0+1, child_containment required, Landlock ABI 6.
      • NRestarts=0.
      • Every command since the deploy completed: 24 of 24, with no ERROR or WARNING records.
      • Each source's first 0.38.0+1 digest equals its last 0.37.0+1 digest:
    info_source_id First fact on 0.38.0+1 output_digest
    01KRYSPTN633JZ8582NHHSAH9A 2026-10-20T04:38:02Z sha256:80ea56dd… unchanged
    01M4RS9CWFZCDNRJXTPTEC6WZ5 2026-10-20T04:50:02Z sha256:f5339c1d… unchanged
    01KTVPAV7RW8TDZH88A43AFT04 2026-10-20T19:35:02Z sha256:8f0f09f6… unchanged
    01KTCNXKXN7Z94G495YEGM6JX3 2026-10-20T19:35:03Z sha256:f77c3471… unchanged
    01KVP9B9HVZ9NK1HRWT5569R62 2026-10-20T22:32:02Z sha256:36e8c9f2… unchanged
    • The Archiver notice was posted, with approval, after the live checks: archiver#355, archiver#356, archiver#378.
      • The #356 note corrects one point in the cannobserv brief there. A jsonpath with no match is processing_complete with empty: true at Processor, not an extraction error.

    Known limits:

    • JSON extraction is not proven live. Every production item is text/html. JSON rests on the synthetic corpus, through the contained child.
    • application/ld+json has no case of its own. It maps to the same JsonExtractor as the two essences that do.
    • ignore_selectors is covered under full_page only, not under css or xpath.

    Follow-ups:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions