Preserve semantic PDF text, Form geometry and bounded OCR evidence

.github/workflows/ci.yml

@@ -140,7 +140,12 @@ jobs:
           node-version: "22"
 
       - name: Test public JavaScript adapters against current CLI
-        run: node --test integrations/javascript/*.test.mjs
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y tesseract-ocr tesseract-ocr-eng poppler-utils
+          tesseract --version
+          pdftoppm -v
+          node --test integrations/javascript/*.test.mjs
 
   python:
     name: Python ${{ matrix.python-version }} bindings and wheel

README.md

@@ -669,7 +669,7 @@ and ruling lines, invokes OCR only for scanned routes, then reconciles native,
 OCR, table, formula, and form spans with typed provenance.
 
 The versioned JSON, artifact JSONL, and streaming JSONL schema is currently
-`0.15.0`, with parser build identity `0.4.0-dev` after the `v0.3.0` release.
+`0.17.0`, with parser build identity `0.4.0-dev` after the `v0.3.0` release.
 Every emitted record carries a `provenance` envelope with document and
 source identity, input hash context, artifact id, page/bbox, source kind,
 confidence, related span/block/chunk ids, route trace ids, and route reasons.
@@ -683,7 +683,15 @@ UTF-16 locators, overlap-aware chunk origins and separate prompt projections.
 Schema `0.15.0` adds reconciled span/table/cell ownership and page/region recovery
 assessments. Streaming supports output and adaptive page allocation budgets;
 the JavaScript page adapter validates lifecycle completion and applies
-backpressure. Visual completeness remains unverified.
+backpressure. Schema `0.16.0` retains rendering-mode and rectangular-clipping
+evidence on native spans and adds bounded, page/region-scoped OCR crops with
+accepted PNG hashes. Crops preserve native text on refusal and expose unresolved
+quota skips. Visual completeness remains unverified; paint evidence describes
+operators and span bounds rather than final pixel visibility.
+Schema `0.17.0` adds content-stream and Form invocation scope to spans.
+ActualText and classified text boundaries survive adaptive reconstruction; Form
+text uses scoped resources and transformed geometry. Native and adaptive text
+quality gates run against the same born-digital and hard-font floors.
 Host adoption remains explicit; accepted schema families and parser/compiler
 pins must be updated together.
 

benchmark/eval/README.md

@@ -582,8 +582,15 @@ pdf-parser benchmark \
 ```
 
 Checked-in homogeneous gates live under `benchmark/eval/gates/`. CI runs the
-born-digital and hard-font gates alongside the broad tiny-corpus scorecard;
-table, OCR, render, and encrypted-twin checks retain their specialist
+born-digital and hard-font gates in both `pdf-parser:native` and
+`pdf-parser:adaptive` lanes alongside the broad tiny-corpus scorecard. Both lanes
+use the same absolute text floors; successful record parsing is a separate check.
+The JavaScript fidelity tests additionally verify batch/stream artifacts,
+semantic source text, chunks and locators for ActualText, writing boundaries and
+resource-scoped Forms. Run `node --test integrations/javascript/*.test.mjs` after
+building the CLI; skipped real-CLI tests do not qualify this boundary.
+
+Table, OCR, render, and encrypted-twin checks retain their specialist
 comparators because their success contracts include structure, routing, pixels,
 or twin equivalence rather than text metrics alone.
 

build.zig

@@ -437,6 +437,28 @@ pub fn build(b: *std.Build) void {
     });
     benchmark_eval_step.dependOn(&benchmark_font_quality_cmd.step);
 
+    // Native decoding and adaptive reconstruction are separate quality lanes.
+    // Both must satisfy the same text floors before public artifacts qualify.
+    const benchmark_adaptive_quality_cmd = b.addRunArtifact(exe);
+    benchmark_adaptive_quality_cmd.step.dependOn(b.getInstallStep());
+    benchmark_adaptive_quality_cmd.addArgs(&.{
+        "benchmark",            "--manifest",                 "benchmark/eval/gates/born-digital.tsv",
+        "--suite-id",           "adaptive-born-digital-gate", "--tools",
+        "pdf-parser:adaptive",  "--thresholds",               "benchmark/eval/gates/born-digital-thresholds.json",
+        "--fail-on-regression", "--fail-on-skipped",
+    });
+    benchmark_eval_step.dependOn(&benchmark_adaptive_quality_cmd.step);
+
+    const benchmark_adaptive_font_quality_cmd = b.addRunArtifact(exe);
+    benchmark_adaptive_font_quality_cmd.step.dependOn(b.getInstallStep());
+    benchmark_adaptive_font_quality_cmd.addArgs(&.{
+        "benchmark",            "--manifest",              "benchmark/eval/gates/hard-fonts.tsv",
+        "--suite-id",           "adaptive-hard-font-gate", "--tools",
+        "pdf-parser:adaptive",  "--thresholds",            "benchmark/eval/gates/hard-fonts-thresholds.json",
+        "--fail-on-regression", "--fail-on-skipped",
+    });
+    benchmark_eval_step.dependOn(&benchmark_adaptive_font_quality_cmd.step);
+
     const test_step = b.step("test", "Run unit tests");
     test_step.dependOn(&run_lib_unit_tests.step);
     test_step.dependOn(&run_simd_unit_tests.step);

docs/host-adapters.md

@@ -74,7 +74,31 @@ asynchronous I/O, supplies the input SHA-256 as source identity, uses a private
 0600 temporary PDF and removes scratch files on completion/failure. Default
 limits are 45 seconds and 25 MiB. OCR is disabled by default;
 `enableOcr:true` uses already-configured local OCR tools. Cancellation, timeout,
-oversized output and process failures remain explicit.
+oversized output and process failures remain explicit. Specialist failure rejects
+the result by default; `allowPartial:true` explicitly permits retaining partial
+artifacts and unresolved recovery assessments.
+
+Both batch and streaming options accept positive integer OCR budgets:
+
+```js
+const options = {
+  binaryPath,
+  enableOcr: true,
+  allowPartial: true,
+  ocrLimits: {
+    maxImagePixels: 40_000_000,
+    maxImageBytes: 32 * 1024 * 1024,
+    maxRegionsPerPage: 8,
+    maxAttemptsPerPage: 16,
+  },
+};
+```
+
+These forward `--ocr-max-image-pixels`, `--ocr-max-image-bytes`,
+`--ocr-max-regions` and `--ocr-max-attempts`. Caps bound accepted rasters and
+per-page invocation counts; child RSS and temporary disk growth are outside
+these budgets. Crop records retain actual PDF-space bounds, accepted PNG SHA-256
+and byte length. Rotation is explicitly unsupported for OCR crops.
 
 The result retains the manifest, raw JSONL, all protocol records, spans, blocks
 and parser chunks. Its `sourceText` is exact block text in artifact order joined
@@ -105,7 +129,7 @@ text hash/view and the record ids; do not resolve them against another run by
 phrase alone.
 
 `parseArtifactJsonl(rawJsonl, {inputSha256, expectedSourceId})` validates manifest-first batch
-ordering, schema `0.14.x`/`0.15.x`, input/provenance identity, text hashes, range boundaries,
+ordering, schema `0.14.x`/`0.15.x`/`0.16.x`/`0.17.x`, input/provenance identity, text hashes, range boundaries,
 full source-map coverage, referenced artifact sets and page relationships. It
 rejects truncated/malformed streams, forged synthetic text and mismatched source
 views. This is a batch reader; use `extractPdfPages` for lifecycle streams.
@@ -115,15 +139,15 @@ views. This is a batch reader; use `extractPdfPages` for lifecycle streams.
 extraction adapter always binds it to the input hash supplied to the CLI.
 For `0.14.x` only, chunks can carry an internal label when manifest/provenance
 source identity is null. The reader retains that label as metadata; it cannot
-override manifest, input hash or provenance identity. Current `0.15.x` records
+override manifest, input hash or provenance identity. Current `0.15.x`–`0.17.x` records
 must agree with the manifest, including null when no source id was supplied.
 
 Specialist records retain their legacy shared provenance labels; uniqueness is
 checked with their actual request/attempt/response/result ids. Some current table
 provenance block references describe pre-reconciliation indices. The adapter
 preserves those raw values and reports missing references as
 `unresolvedProvenanceLinks`, rather than using them for source-text locators.
-Core span/block/chunk maps remain strict. In schema `0.15.x`, `document.tables`
+Core span/block/chunk maps remain strict. In schema `0.15.x`–`0.17.x`, `document.tables`
 exposes table/cell `reconciled_block_ids` checked against owning source spans.
 Span `reconciled_block_id` and ownership provenance must agree with block maps;
 cells must share their table's page. Use these authoritative fields for links.
@@ -148,8 +172,26 @@ Schema `0.15.x` adds `document.pageOutcomes` and page/region route `coverage`.
 These distinguish retained text, empty unverified output, required/failed
 recovery and recovered regions. The reader rejects out-of-document pages and
 contradictory recovery states. Preserve unresolved assessments in intake and
-retrieval policy. Paint/clipping evidence and regional OCR crops remain pending;
-these assessments do not establish quote verification or publishing authority.
+retrieval policy. Schema `0.16.x` adds span `paint`, three paint counters and
+page/region-scoped OCR crop evidence. The reader rejects inconsistent selected
+attempts, crop budgets and specialist context links. Paint describes operators
+and bbox clipping, with unknown states for unqualified cases; it does not prove
+pixel visibility. These assessments do not establish quote verification or
+publishing authority.
+
+Schema `0.17.x` adds nullable span `content_scope`. A known scope carries its
+page, PDF stream object reference and per-page Form invocation index. Repeated
+Form placements remain distinguishable even when they reuse local font names or
+MCIDs. Coincident identical text may still merge during reconciliation: public
+scope identifies the retained occurrence, while raw glyph inventory distinguishes
+the contributing invocations. The adapter rejects out-of-page or malformed scopes and inconsistent
+specialist context. A locator's `sourceSpanId` resolves to the retained span and
+its scope; null scope explicitly withholds that attribution.
+
+ActualText replacements are preserved once per scope in `sourceText` and parser
+chunks. Unequal glyph/character counts retain span-bbox precision; no exact
+character geometry is inferred from a semantic replacement. Explicit/inferred
+separators survive reconstruction, with synthetic source-map gaps retained.
 
 ## Incremental pages and resource budgets
 
@@ -191,8 +233,8 @@ as `memory_limit`/`output_limit`, without a complete receipt.
 ## Adoption and verification
 
 Readers limited to `0.11.x`/`0.13.x` must add
-`0.15.x` with contract tests before changing their parser pin. The adapter retains
-`0.14.x` reader support. Applications must pair
+`0.17.x` with contract tests before changing their parser pin. The adapter retains
+`0.14.x`, `0.15.x` and `0.16.x` reader support. Applications must pair
 their adopted parser revision with the selected Zig 0.17/compiler/archive identities
 and run its baseline CPU/runtime smoke lane. Formula children must echo the new
 request schema version. Neither host is changed by this work.
@@ -211,3 +253,7 @@ It reports a skip when no built executable exists; that skip does not qualify
 the integration. Unit tests cover Unicode byte/unit translation, repeated
 phrases, zero/large overlap, forged mappings, process limits, cancellation and
 scratch cleanup. No host database, browser, deployment or paid model is exercised.
+The fidelity regressions also exercise ActualText, Unicode, font/paint changes,
+rotated word boundaries, Form resource collisions and repeated placements through
+batch/stream source text and locators. Forms are tested with external text tools
+both present and unavailable.

docs/output-schema.md

@@ -6,10 +6,10 @@ should depend on the records documented here.
 
 ## Version
 
-Current schema version: `0.15.0`
+Current schema version: `0.17.0`
 
 Current parser build identity: `0.4.0-dev`. The `v0.3.0` release emitted schema
-`0.11.0`; the development suffix keeps post-release schema `0.15.0` artifacts
+`0.11.0`; the development suffix keeps post-release schema `0.17.0` artifacts
 distinguishable until the next release is tagged.
 
 The project is still pre-`1.0.0`, so incompatible schema changes may happen.
@@ -34,6 +34,18 @@ JSON output is a `document_manifest` object with arrays of typed records.
 but records are produced page by page so host applications can persist partial
 artifacts and enqueue embeddings before the document finishes.
 
+Schema `0.17.0` adds nullable `content_scope` to spans and inline specialist span
+context. It identifies native page content or a particular Form invocation.
+Semantic replacements and inferred text boundaries survive adaptive
+reconstruction; the existing text-view and source-map contract remains in place.
+Readers must explicitly accept `0.17.x`; formula children must echo it.
+
+Schema `0.16.0` adds span paint evidence, coverage paint counts, and bounded
+regional OCR attempts with accepted crop identity and execution budgets.
+Specialist span/block context uses document-global IDs in both batch and stream
+output. Readers must explicitly accept `0.16.x`; formula children must echo it.
+These additions retain extracted text and keep visual completeness unverified.
+
 Schema `0.15.0` adds authoritative reconciled block references to spans, tables
 and cells, and page/region recovery assessments to route traces. Span provenance
 block references now resolve to actual reconciled owners, including later pages
@@ -253,6 +265,38 @@ failures from recognize failures.
 successful candidate is selected deterministically by outcome, confidence,
 coverage, character count, then lower DPI and attempt index.
 
+In `0.16`, OCR attempts have `page_index` and nullable `region_index`. IDs use
+`specialist-attempt-ocr-page-P-I` for whole pages and
+`specialist-attempt-ocr-page-P-region-R-I` for crops; response/result IDs use the
+same page/region scope. Selection, counts and recovery bind to that exact scope.
+A response describes the last attempt; its selected winner may be an earlier
+completed attempt after a failed retry. Completed results require a completed
+selected OCR attempt.
+Compatibility OCR responses summarize timeout/invalid output as `failed`;
+the attempt retains the detailed `timeout`/`invalid_output` status and diagnostics.
+
+OCR `config` also records `max_image_pixels`, `max_image_bytes`,
+`max_regions_per_page` and `max_attempts_per_page`. Defaults are 40,000,000
+pixels, 33,554,432 bytes, 8 regions and 16 attempts per page. Whole-page attempts
+have `crop:null`; regional attempts carry `crop` with `bbox`, `dpi`,
+`pixel_width`, `pixel_height`, nullable `image_sha256` and `image_byte_length`.
+The hash and length identify the accepted PNG bytes delivered to recognition.
+They stay null when rasterization is refused or fails. The bbox is the actual
+pixel-aligned crop in PDF user space: padded by two points, clamped to MediaBox,
+then rounded outward using the nonzero page origin. Ephemeral PNG paths are
+deleted after each attempt and never exposed as public crop identity.
+Whole-page OCR preserves its established MediaBox coordinate mapping; crop
+rounding does not rescale whole-page recognition geometry.
+
+Good native pages route image placements individually; pages failing native
+quality use whole-page recovery. Native text survives failed recovery. Quota
+skips retain requested routes and `not_invoked` attempts with `ocr_region_limit`
+or `ocr_attempt_limit`. Pixel/byte refusal, invalid geometry and rotation have
+explicit diagnostics. Rotated OCR pages are refused until their raster mapping
+is qualified. Pixel bounds apply before launch and to accepted PNG dimensions;
+byte bounds apply before reading the PNG. These are not child RSS or in-flight
+temporary-file disk caps.
+
 ### `specialist_response`
 
 Response record from a specialist boundary. Existing Tesseract OCR output is
@@ -370,8 +414,74 @@ page; retained native text does not prove that image regions were recognized.
 Selected OCR/formula attempts count as recovery only when retained recognized
 spans intersect the assessed region. Successful OCR cannot hide unresolved
 formula or disjoint image regions. Lifecycle `completed` means extraction
-finished, not that the source has complete visual coverage. Paint/clipping
-evidence and bounded regional OCR crops remain future work.
+finished, not that the source has complete visual coverage.
+
+### Paint and clipping evidence (0.16)
+
+Spans and inline specialist span context include `paint`:
+
+```json
+{"rendering_mode":0,"paint_state":"painting","clip_state":"inside","clip_bbox":{"x0":0,"y0":0,"x1":300,"y1":300}}
+```
+
+`rendering_mode` is 0–7 or null. `paint_state` is `painting`, `non_painting` or
+`unknown`; modes 3 and 7 are nonpainting. Type 3 fonts retain unknown paint
+state for the other modes. `clip_state` is `unclipped`, `inside`, `outside`,
+`partial` or `unknown`. Known rectangular clips carry their PDF-space bbox.
+Span geometry follows the current affine transform, with graphics state restored
+by q/Q. Text clipping takes effect after the text object; arbitrary paths,
+multiple subpaths and uncertain transforms retain unknown clip evidence.
+
+This is operator and bbox evidence, not final pixel visibility. It does not
+qualify opacity, transparency, glyph outlines, recursive Form bounds or the
+initial page clip. Nonpainting/clipped text is retained as native source text.
+Page/region `coverage` adds `non_painting_span_count`, `clipped_out_span_count`
+and `uncertain_paint_span_count`; these counts never establish completeness.
+
+### Semantic text and content scope (0.17)
+
+`content_scope` is null when a single native scope is unknown or not retained by
+a derived artifact. Native page content has the following scope:
+
+```json
+{"page_index":0,"stream_object_num":null,"stream_generation":0,"invocation_index":0}
+```
+
+A Form scope carries its PDF object number/generation and a positive invocation
+index assigned in traversal order on that page. Repeated placements of the same
+Form share the object reference but have distinct invocation indices. Scope is
+bound to the input identity and page; it does not establish logical reading order.
+Reconciliation may merge coincident identical text, retaining the chosen span's
+scope and a duplicate count. Raw glyph inventory preserves invocation identity;
+public spans do not enumerate every contributing invocation scope.
+MCIDs are interpreted within their content stream, not as page-global identities.
+Fallback/OCR and reconstructed table text may have null scope. Specialist span
+context must agree with the corresponding public span's scope.
+
+ActualText replacement strings are authoritative semantic text. PDF text strings
+decode UTF-16BE with a BOM or PDFDocEncoding, including extended punctuation,
+accents, ligatures and Euro. Undefined PDFDocEncoding bytes become U+FFFD.
+Named marked-content properties resolve through the current resource scope.
+The native
+geometry API retains original glyph evidence separately from the semantic view;
+adaptive spans, blocks, chunks and text views consume the semantic view. Each
+replacement is emitted once over its marked-content scope, including across font,
+paint and text-operator changes. Empty replacement strings suppress that scope's
+raw text. A differing glyph/character count uses union geometry, not fabricated
+per-character placement; conflicting paint or content-scope evidence stays unknown.
+
+For conflicting nested ActualText declarations, the parser uses an explicit
+outer-owner policy: the enclosing replacement owns its entire lexical scope.
+This is a deterministic parser policy, not a claim of agreement among external
+extractors. Source locators retain span-bbox precision. Search results for an
+aggregate semantic replacement may have null raw glyph indices while retaining
+the union bbox.
+
+Form traversal shares resource resolution across extraction paths, composes the
+Form matrix, and retains Form-bbox clip evidence conservatively. Local font
+resources are scoped independently of page font names. Cycles, nesting beyond ten
+Forms and more than 10,000 Form invocations per page fail explicitly. These are
+traversal bounds, not whole-process memory or visual-completeness guarantees.
 
 ### Streaming resource budgets