Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ jobs:
python3 -m unittest discover -s benchmark/eval -p 'test_table_compare.py'
python3 -m unittest discover -s benchmark/eval -p 'test_bounded_zig.py'
python3 -m unittest discover -s benchmark/eval/table_topology -p 'test_*.py'
python3 -m unittest discover -s benchmark/eval/table_topology_real -p 'test_*.py'
python3 -m unittest discover -s benchmark/eval/reading_order_next -p 'test_*.py'
python3 benchmark/eval/table_topology/generate.py --check
python3 benchmark/eval/reading_order_next/generate.py --check
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@

### Added

- A source-bound real-document table pack freezes selected merged headers and
continuation controls across separate development and holdout issuers. Local
original PDFs stay outside Git; failed qualifications remain evidence.

- Frozen development and holdout suites check table topology, exact cell source
ownership, and emitted reading order independently of graph diagnostics.
- A project-local build monitor bounds observed compiler-cache allocation and
Expand All @@ -20,6 +24,9 @@

### Fixed

- Schema 0.19.2 keeps batch debug assets bound to original document pages when
extraction selects a window starting after page one.

- Schema 0.19.1 keeps reconstructed table evidence at cell granularity, assigns
shared-rule text once, and retains prose surrounding a promoted table.
- Legacy native ordering resets columns at corroborated section headings.
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -670,11 +670,13 @@ and ruling lines, invokes OCR only for scanned routes, then reconciles native,
OCR, table, formula, and form spans with typed provenance.

The versioned JSON, artifact JSONL, and streaming JSONL schema is currently
`0.19.1` in local development, with parser build identity `0.4.0-dev` after the
`0.19.2` in local development, with parser build identity `0.4.0-dev` after the
`v0.3.0` release. Schema `0.17.0` landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6)
at `9cac608`, with all six CI checks green. Schemas `0.18.0` and `0.19.0` landed
in [PR #7](https://github.com/Tom-R-Main/pdf-parser/pull/7) at `478d73a`, with all
six PR CI checks green. Schema `0.19.1` is a local working-tree continuation.
six PR CI checks green. Schema `0.19.1` landed in
[PR #8](https://github.com/Tom-R-Main/pdf-parser/pull/8) at `1483f41`, with all
six PR CI checks green. Schema `0.19.2` is a local working-tree continuation.
The `0.19` vector-routing continuation's 23 outlined-text and
negative-control fixtures, 13 real CLI/SDK tests and both JSONL readers are
qualified in the [implementation evidence](docs/parser-host-implementation-plan.md#bounded-vector-routing-019-locally-checked).
Expand Down
6 changes: 6 additions & 0 deletions benchmark/eval/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Evaluation Harness

The [real-document table pack](table_topology_real/README.md) freezes selected
merged headers and positive/negative continuation checks from three complete
government originals. It preserves failed qualifications and uses local,
hash-verified inputs without CI downloads. Selected-cell checks are not a
complete-table topology accuracy measure.

New frozen [table topology controls](table_topology/README.md) and
[reading-order holdouts](reading_order_next/README.md) assess source ownership,
physical cell spans and emitted order separately. The earlier reading-order
Expand Down
70 changes: 70 additions & 0 deletions benchmark/eval/table_topology_real/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Real-document table regions and continuation

This pack checks selected rows, literal merged header cells, and continuation
links in three complete government publications. It complements the frozen
synthetic topology pack; it does not measure complete-table topology accuracy.

The development source is an NCUA financial report. Treasury cash/debt tables
and Census construction tables are holdouts. Issuers and template families are
disjoint across the split. Source cards bind complete original PDFs by SHA256,
physical page count, source URL, and the reference checkout commit. Rights
metadata is retained from those cards; no agency endorses this evaluation.

Independent Poppler text, word geometry, and source renders established truth
before parser assessment. Rendered rules establish Census header spans of two
and three columns, with a two-row stub. The Treasury debt-limit table includes
an integrated caption, two vertically merged headers, and three opening-balance
periods. Approximate raw PDF regions select tables; they do not qualify exact
cell-edge geometry. `freeze.json` binds source cards, selected truth, selectors,
and the blind evidence inventory.

Both public JSONL formats are assessed independently. Missing or ambiguous
tables fail. Negative continuation checks require both tables to exist;
positive links must be reciprocal with a shared logical identity and consecutive
part indices. Row checks preserve currency markers, zeros, dashes, parentheses,
and `NA`, with whitespace normalization only. Treasury prefix checks select the
deposit/issues panel without combining its entity with an adjacent withdrawal
or redemption panel. Header checks require literal placement and spans. Context
notes describe source units and measurement bases; they are not assertions of a
public unit field.

Prepare existing local originals without downloading or overwriting mismatches:

```sh
python3 benchmark/eval/table_topology_real/prepare.py \
--source-root /path/to/originals
python3 -m unittest discover -s benchmark/eval/table_topology_real -p 'test_*.py'
```

The default prepared input directory is ignored
`benchmark/eval/raw_cache/table_topology_real`. Complete source files remain
outside Git. Assess development first, then the frozen holdouts:

```sh
python3 benchmark/eval/table_topology_real/verify.py \
--cli zig-out/bin/pdf-parser --split dev \
--output benchmark/eval/outputs/real-table-dev
python3 benchmark/eval/table_topology_real/verify.py \
--cli zig-out/bin/pdf-parser --split holdout \
--output benchmark/eval/outputs/real-table-holdout
```

Output directories must be new. Each run preserves commands, raw output,
stderr, failures, source/binary/verifier hashes, and a report. A failed
qualification exits nonzero; a failed baseline remains useful evidence.
CI tests the scorer and frozen metadata without fetching the source PDFs.

Only one explicit positive-continuation source/template is represented. The
Treasury Table IIIA/IIIB pair and the Census annualized/unadjusted pair are
negative controls despite repeated headers or amounts. Distinct NCUA statements
are negatives too. Do not infer cross-template positive-continuation accuracy.

Default local Treasury renders failed because unembedded Helvetica substituted
incorrectly. Blind review used an isolated Fontconfig file and existing Arial
fonts, with its cache under the ignored evidence directory; failed renders were
preserved. NCUA footer rendering is excluded from truth.

Retain observed failures. Do not tune selectors, truth, thresholds, or parser
behavior around these holdouts. Correct objective harness defects with
regressions and preserve the original report. Future parser changes need separate
development evidence and fresh holdouts before broader qualification claims.
132 changes: 132 additions & 0 deletions benchmark/eval/table_topology_real/cases.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
[
{
"id": "ncua-statements-dev", "source_id": "e013", "split": "dev", "pages": [3, 5],
"tables": [
{
"id": "balance", "page_index": 2, "bbox": [51, 139, 557, 650],
"context": "Balance sheets, dollars in thousands",
"rows": [
{"tokens": "Fund Balance with Treasury $ 28,485 $ 26,245"},
{"tokens": "Accounts Payable - Due to the NCUA Operating Fund $ - $ 19"},
{"tokens": "TOTAL ASSETS $ 20,553,523 $ 20,483,729"}
]
},
{
"id": "net-cost", "page_index": 3, "bbox": [52, 272, 557, 658],
"context": "Net cost; July, year-to-date July 2023, year-to-date July 2022; thousands",
"rows": [
{"tokens": "Reserve Expense (Reduction) - 20,002 7,506"},
{"tokens": "AME Receivable Bad Debt Expense - (21,778) (51,864)"}
]
},
{
"id": "net-position", "page_index": 4, "bbox": [52, 207, 559, 645],
"context": "Changes in net position; July 2023 and July 2022; thousands",
"rows": [
{"tokens": "Beginning Balances $ 3,258,998 $ 4,780,200"},
{"tokens": "Beginning Balances 16,916,436 15,783,657"},
{"tokens": "NET POSITION $ 20,346,828 $ 20,311,501", "occurrences": 2}
]
}
],
"relations": [
{"from": "balance", "to": "net-cost", "continues": false},
{"from": "net-cost", "to": "net-position", "continues": false}
]
},
{
"id": "treasury-continuation-holdout", "source_id": "e012", "split": "holdout", "pages": [1, 3],
"tables": [
{
"id": "cash-1", "page_index": 0, "bbox": [36.8, 31.5, 581.1, 589.5], "column_count": 8,
"context": "Table II: independent deposit and withdrawal panels, millions",
"rows": [
{"tokens": "Dept of Agriculture (USDA) - misc 30 273 1,904", "prefix": true},
{"tokens": "USDA - Federal Crop Insurance Corp Fund 0 0 33", "prefix": true},
{"tokens": "Legislative Branch - misc 0 3 40", "prefix": true}
]
},
{
"id": "cash-2", "page_index": 1, "bbox": [36.8, 341, 581.1, 735.7], "column_count": 8,
"context": "Table II - cont.; blank deposit detail region beside continuing withdrawals",
"rows": [
{"tokens": "Legislative Branch - Library of Congress 0 38 146", "prefix": true},
{"tokens": "Unclassified - Deposits 10 217 2,892", "prefix": true},
{"tokens": "Total Deposits 8,170 408,982 2,621,282", "prefix": true},
{"tokens": "Treasury General Account Total Deposits $ 224,427 $ 2,155,471 $ 17,629,416", "prefix": true}
]
},
{
"id": "debt-face-value", "page_index": 1, "bbox": [36.8, 131.5, 581.1, 332], "column_count": 8,
"context": "Table IIIA public debt transactions, face value, millions",
"rows": [
{"tokens": "Government Account Series 570,559 7,051,129 67,726,222", "prefix": true},
{"tokens": "Total Issues $ 788,818 $ 8,823,334 $ 82,899,841", "prefix": true}
]
},
{
"id": "debt-cash-basis", "page_index": 2, "bbox": [36.8, 570.5, 581.1, 735.7], "column_count": 8,
"context": "Table IIIB adjustment to cash basis; repeated amounts are not continuation",
"rows": [
{"tokens": "Public Debt Issues (Table IIIA) $ 788,818 $ 8,823,334 $ 82,899,841", "prefix": true},
{"tokens": "Government Account Transactions (-) 570,559 7,051,129 67,726,222", "prefix": true}
]
},
{
"id": "debt-limit-groups", "page_index": 2, "bbox": [36.8, 408.5, 581.1, 560.5], "column_count": 5,
"context": "Table IIIC: integrated caption; closing balance and opening balance periods, millions",
"rows": [
{"tokens": "Debt Held by the Public $ 28,900,195 $ 28,924,758 $ 28,914,411 $ 28,307,312"},
{"tokens": "Intragovernmental Holdings 7,315,899 7,290,501 7,304,513 7,157,362"}
],
"headers": [
{"text": "TABLE IIIC - Debt Subject to Limit", "row": 0, "column": 0, "rowspan": 1, "colspan": 5},
{"text": "Balance Transactions", "row": 1, "column": 0, "rowspan": 2, "colspan": 1},
{"text": "Closing balance today", "row": 1, "column": 1, "rowspan": 2, "colspan": 1},
{"text": "Opening balance", "row": 1, "column": 2, "rowspan": 1, "colspan": 3},
{"text": "Today", "row": 2, "column": 2, "rowspan": 1, "colspan": 1},
{"text": "This month", "row": 2, "column": 3, "rowspan": 1, "colspan": 1},
{"text": "Fiscal year", "row": 2, "column": 4, "rowspan": 1, "colspan": 1}
]
}
],
"relations": [
{"from": "cash-1", "to": "cash-2", "continues": true},
{"from": "debt-face-value", "to": "debt-cash-basis", "continues": false}
]
},
{
"id": "construction-groups-holdout", "source_id": "e019", "split": "holdout", "pages": [3, 5],
"tables": [
{
"id": "annualized", "page_index": 2, "bbox": [40.5, 133.5, 544, 732], "column_count": 9,
"context": "Table 1 seasonally adjusted annual rate, millions",
"rows": [{"tokens": "Total Construction 2,196,060 2,206,880 2,194,337 2,191,059 2,184,796 2,135,771 -0.5 2.8"}]
},
{
"id": "unadjusted", "page_index": 3, "bbox": [40.5, 138.5, 559, 732], "column_count": 10,
"context": "Table 2 not seasonally adjusted monthly values, millions",
"rows": [{"tokens": "Total Construction 172,187 157,055 156,483 164,364 178,806 167,395 485,726 472,183 2.9"}]
},
{
"id": "variation", "page_index": 4, "bbox": [40.5, 110.5, 558, 729.7], "column_count": 6,
"context": "Table 3 percent; NA means Not applicable, not zero or blank",
"rows": [
{"tokens": "Total Construction 0.7 0.5 0.6 0.5 0.7"},
{"tokens": "Communication NA NA NA NA NA", "occurrences": 2}
],
"headers": [
{"text": "Type of Construction", "row": 0, "column": 0, "rowspan": 2, "colspan": 1},
{"text": "Coefficient of variation", "row": 0, "column": 1, "rowspan": 1, "colspan": 2},
{"text": "Standard error", "row": 0, "column": 3, "rowspan": 1, "colspan": 3},
{"text": "Monthly estimate", "row": 1, "column": 1, "rowspan": 1, "colspan": 1},
{"text": "Year-to-date estimate", "row": 1, "column": 2, "rowspan": 1, "colspan": 1},
{"text": "Year-to-date change", "row": 1, "column": 3, "rowspan": 1, "colspan": 1},
{"text": "Month-to-month change", "row": 1, "column": 4, "rowspan": 1, "colspan": 1},
{"text": "Month-to-month change from prev. year", "row": 1, "column": 5, "rowspan": 1, "colspan": 1}
]
}
],
"relations": [{"from": "annualized", "to": "unadjusted", "continues": false}]
}
]
28 changes: 28 additions & 0 deletions benchmark/eval/table_topology_real/freeze.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"schema_version": 1,
"frozen_at": "2026-10-12T19:13:32.616875+00:00",
"parser_outputs_observed": false,
"annotation_method": "Independent Poppler layout text, word geometry and visually reviewed source renders; government originals unchanged. Approximate raw PDF regions; literal selected cell/header facts only.",
"files_sha256": {
"sources.json": "f037e47aeeddc5def583ccd6678703ed6471668333c126cdba3ce614d1a73fc5",
"cases.json": "803d98a043407faaa3fa647de9fc4b435fc8efe730c2de17d6f0054f5f111147"
},
"controls": {
"min_region_coverage": 0.7,
"min_table_coverage": 0.7
},
"blind_evidence_sha256": {
"blind-annotation-suggestions.json": "db11fd1700c3f7ca0cea913cc6868b8f28395b91450e7a91e5b959996d2ffd3f",
"blind-annotation-supplement-table-iiic.json": "11f2cfe5b33ee4613f54ca14c5461815e2816a2d9ca6f32cec295897fe8ad318",
"source-identity-audit.json": "a33bfdc5f61a36235c2a1943a2a127606d5c5930b50e6c5602300662aa33914b",
"source-review-evidence-inventory.json": "6fb3745ec30f3ede4f7ccd86e122cfebadbf30ce2aef4798b4a3c59f56d72a72"
},
"split_policy": "Issuer and template disjoint: NCUA development; Treasury and Census holdout. No parser tuning after holdout assessment.",
"limits": [
"Partial selected-cell and region truth; no complete-table topology metric.",
"One positive-continuation source/template; no cross-template positive generalization.",
"No exact cell-edge geometry accuracy claim.",
"Default Treasury renders failed due to unembedded Helvetica substitution; source-review includes readable isolated existing-Arial renders and failed originals."
],
"stop_rule": "Retain failing baseline; do not modify truth, selectors, thresholds or kernel to fit observed holdouts. Correct objective harness defects with regressions and preserved initial evidence; use separate development data for future parser changes."
}
77 changes: 77 additions & 0 deletions benchmark/eval/table_topology_real/prepare.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
"""Copy hash-verified local originals into the ignored evaluation source cache."""
import argparse
import json
import os
from pathlib import Path
import tempfile

from verify import DEFAULT_SOURCE_ROOT, ROOT, basename, load_corpus, sha


def prepare_sources(source_root, output, sources):
source_root, output = Path(source_root), Path(output)
if source_root.resolve() == output.resolve():
raise ValueError("source and destination must differ")
if not isinstance(sources, list) or not sources:
raise ValueError("empty source inventory")
names = set()
# Check the complete inventory before copying anything.
for source in sources:
name = source.get("filename")
if not basename(name) or name in names:
raise ValueError("invalid or duplicate source basename")
names.add(name)
original = source_root / name
if original.is_symlink() or not original.is_file() or sha(original) != source["sha256"]:
raise ValueError("wrong local source hash: " + name)
destination = output / name
if destination.is_symlink() or (destination.exists() and (not destination.is_file() or sha(destination) != source["sha256"])):
raise ValueError("refusing to replace destination: " + name)
output.mkdir(parents=True, exist_ok=True)
result = []
for source in sources:
name = source["filename"]
original, destination = source_root / name, output / name
reused = destination.exists()
if not reused:
temporary = None
try:
with tempfile.NamedTemporaryFile(prefix=".prepare-", dir=output, delete=False) as handle:
temporary = Path(handle.name)
with original.open("rb") as input_file:
for chunk in iter(lambda: input_file.read(1024 * 1024), b""):
handle.write(chunk)
handle.flush()
os.fsync(handle.fileno())
if sha(temporary) != source["sha256"] or sha(original) != source["sha256"]:
raise ValueError("source changed while preparing: " + name)
# Atomic no-clobber publication; never replace an existing file.
try:
os.link(temporary, destination)
except FileExistsError:
if destination.is_symlink() or not destination.is_file() or sha(destination) != source["sha256"]:
raise ValueError("destination appeared with wrong hash: " + name)
reused = True
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
if sha(original) != source["sha256"] or sha(destination) != source["sha256"]:
raise ValueError("post-copy source/destination hash mismatch: " + name)
result.append(dict(id=source["id"], filename=name, sha256=source["sha256"], reused=reused))
return result


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-root", type=Path, required=True)
parser.add_argument("--output", type=Path, default=DEFAULT_SOURCE_ROOT)
args = parser.parse_args()
sources, _, _ = load_corpus(ROOT)
result = prepare_sources(args.source_root, args.output, sources)
print(json.dumps(dict(status="prepared", output=str(args.output), sources=result), indent=2))
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading