From 376d2d928fc62f121422570ea25c366e26409560 Mon Sep 17 00:00:00 2001 From: Tom Main Date: Sun, 11 Oct 2026 16:22:29 -0400 Subject: [PATCH 1/2] Add coordinate-aware vector OCR routing and public evidence --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 34 ++ README.md | 43 +- benchmark/eval/vector_routing/.gitignore | 4 + benchmark/eval/vector_routing/border.pdf | Bin 0 -> 678 bytes .../eval/vector_routing/chart-labels.pdf | Bin 0 -> 7878 bytes benchmark/eval/vector_routing/clipped-out.pdf | Bin 0 -> 7956 bytes .../eval/vector_routing/compound-fill.pdf | Bin 0 -> 7907 bytes .../eval/vector_routing/dense-graphics.pdf | Bin 0 -> 1636 bytes benchmark/eval/vector_routing/filled-box.pdf | Bin 0 -> 679 bytes .../eval/vector_routing/font-license.txt | 57 ++ benchmark/eval/vector_routing/fused-panel.pdf | Bin 0 -> 7978 bytes .../eval/vector_routing/generate_fixtures.py | 291 ++++++++++ .../eval/vector_routing/image-overlap.pdf | Bin 0 -> 8620 bytes benchmark/eval/vector_routing/manifest.json | 468 ++++++++++++++++ .../vector_routing/mixed-native-outlines.pdf | Bin 0 -> 7933 bytes .../eval/vector_routing/native-highlight.pdf | Bin 0 -> 681 bytes .../eval/vector_routing/outlined-text.pdf | Bin 0 -> 7865 bytes .../vector_routing/partial-native-clip.pdf | Bin 0 -> 7972 bytes .../vector_routing/quarter-turn-outlines.pdf | Bin 0 -> 7899 bytes .../vector_routing/repeated-multipage.pdf | Bin 0 -> 29417 bytes .../vector_routing/rule-through-outlines.pdf | Bin 0 -> 7960 bytes benchmark/eval/vector_routing/segment-cap.pdf | Bin 0 -> 5888 bytes benchmark/eval/vector_routing/shape-cap.pdf | Bin 0 -> 11699 bytes .../eval/vector_routing/sheared-form.pdf | Bin 0 -> 7728 bytes benchmark/eval/vector_routing/single-logo.pdf | Bin 0 -> 757 bytes .../vector_routing/stroked-letterforms.pdf | Bin 0 -> 7939 bytes .../eval/vector_routing/thin-filled-rule.pdf | Bin 0 -> 673 bytes .../eval/vector_routing/thin-stroked-rule.pdf | Bin 0 -> 682 bytes .../eval/vector_routing/transformed-form.pdf | Bin 0 -> 7722 bytes .../eval/vector_routing/truth/border.txt | 1 + .../vector_routing/truth/chart-labels.txt | 3 + .../eval/vector_routing/truth/clipped-out.txt | 1 + .../vector_routing/truth/compound-fill.txt | 2 + .../vector_routing/truth/dense-graphics.txt | 1 + .../eval/vector_routing/truth/filled-box.txt | 1 + .../eval/vector_routing/truth/fused-panel.txt | 3 + .../vector_routing/truth/image-overlap.txt | 2 + .../truth/mixed-native-outlines.txt | 2 + .../vector_routing/truth/native-highlight.txt | 1 + .../vector_routing/truth/outlined-text.txt | 1 + .../truth/partial-native-clip.txt | 2 + .../truth/quarter-turn-outlines.txt | 2 + .../truth/repeated-multipage.txt | 6 + .../truth/rule-through-outlines.txt | 2 + .../eval/vector_routing/truth/segment-cap.txt | 1 + .../eval/vector_routing/truth/shape-cap.txt | 1 + .../vector_routing/truth/sheared-form.txt | 2 + .../eval/vector_routing/truth/single-logo.txt | 1 + .../truth/stroked-letterforms.txt | 2 + .../vector_routing/truth/thin-filled-rule.txt | 1 + .../truth/thin-stroked-rule.txt | 1 + .../vector_routing/truth/transformed-form.txt | 2 + docs/host-adapters.md | 91 +++- docs/output-schema.md | 142 ++++- docs/parser-host-implementation-plan.md | 126 ++++- .../coordinates-integration.test.mjs | 198 +++++++ integrations/javascript/coordinates.d.mts | 32 ++ integrations/javascript/coordinates.mjs | 135 +++++ integrations/javascript/coordinates.test.mjs | 140 +++++ integrations/javascript/extract.d.mts | 72 +++ integrations/javascript/extract.mjs | 110 +++- integrations/javascript/extract.test.mjs | 121 +++++ .../javascript/fixtures/quarter-scan.jpg | Bin 0 -> 38165 bytes integrations/javascript/index.d.mts | 1 + integrations/javascript/index.mjs | 1 + integrations/javascript/integration.test.mjs | 4 +- integrations/javascript/stream.mjs | 2 +- integrations/javascript/stream.test.mjs | 45 ++ .../javascript/vector-contract.test.mjs | 259 +++++++++ .../javascript/vector-routing.test.mjs | 321 +++++++++++ src/adaptive.zig | 97 +++- src/coverage.zig | 30 +- src/eval_runner.zig | 19 +- src/integration_test.zig | 78 ++- src/main.zig | 23 +- src/ocr.zig | 18 + src/ocr/crop.zig | 33 +- src/ocr/tesseract_c.zig | 65 ++- src/ocr/tesseract_cli.zig | 45 +- src/ocr/tesseract_tsv.zig | 167 +++++- src/ocr/types.zig | 54 ++ src/page_geometry.zig | 505 ++++++++++++++++++ src/pagetree.zig | 109 ++++ src/paint.zig | 15 +- src/reconcile.zig | 54 +- src/root.zig | 410 +++++++++++--- src/schema.zig | 45 ++ src/specialist_protocol.zig | 50 +- src/specialists.zig | 5 + src/vector_evidence.zig | 482 +++++++++++++++++ src/vector_routing.zig | 317 +++++++++++ 92 files changed, 5129 insertions(+), 232 deletions(-) create mode 100644 CHANGELOG.md create mode 100644 benchmark/eval/vector_routing/.gitignore create mode 100644 benchmark/eval/vector_routing/border.pdf create mode 100644 benchmark/eval/vector_routing/chart-labels.pdf create mode 100644 benchmark/eval/vector_routing/clipped-out.pdf create mode 100644 benchmark/eval/vector_routing/compound-fill.pdf create mode 100644 benchmark/eval/vector_routing/dense-graphics.pdf create mode 100644 benchmark/eval/vector_routing/filled-box.pdf create mode 100644 benchmark/eval/vector_routing/font-license.txt create mode 100644 benchmark/eval/vector_routing/fused-panel.pdf create mode 100644 benchmark/eval/vector_routing/generate_fixtures.py create mode 100644 benchmark/eval/vector_routing/image-overlap.pdf create mode 100644 benchmark/eval/vector_routing/manifest.json create mode 100644 benchmark/eval/vector_routing/mixed-native-outlines.pdf create mode 100644 benchmark/eval/vector_routing/native-highlight.pdf create mode 100644 benchmark/eval/vector_routing/outlined-text.pdf create mode 100644 benchmark/eval/vector_routing/partial-native-clip.pdf create mode 100644 benchmark/eval/vector_routing/quarter-turn-outlines.pdf create mode 100644 benchmark/eval/vector_routing/repeated-multipage.pdf create mode 100644 benchmark/eval/vector_routing/rule-through-outlines.pdf create mode 100644 benchmark/eval/vector_routing/segment-cap.pdf create mode 100644 benchmark/eval/vector_routing/shape-cap.pdf create mode 100644 benchmark/eval/vector_routing/sheared-form.pdf create mode 100644 benchmark/eval/vector_routing/single-logo.pdf create mode 100644 benchmark/eval/vector_routing/stroked-letterforms.pdf create mode 100644 benchmark/eval/vector_routing/thin-filled-rule.pdf create mode 100644 benchmark/eval/vector_routing/thin-stroked-rule.pdf create mode 100644 benchmark/eval/vector_routing/transformed-form.pdf create mode 100644 benchmark/eval/vector_routing/truth/border.txt create mode 100644 benchmark/eval/vector_routing/truth/chart-labels.txt create mode 100644 benchmark/eval/vector_routing/truth/clipped-out.txt create mode 100644 benchmark/eval/vector_routing/truth/compound-fill.txt create mode 100644 benchmark/eval/vector_routing/truth/dense-graphics.txt create mode 100644 benchmark/eval/vector_routing/truth/filled-box.txt create mode 100644 benchmark/eval/vector_routing/truth/fused-panel.txt create mode 100644 benchmark/eval/vector_routing/truth/image-overlap.txt create mode 100644 benchmark/eval/vector_routing/truth/mixed-native-outlines.txt create mode 100644 benchmark/eval/vector_routing/truth/native-highlight.txt create mode 100644 benchmark/eval/vector_routing/truth/outlined-text.txt create mode 100644 benchmark/eval/vector_routing/truth/partial-native-clip.txt create mode 100644 benchmark/eval/vector_routing/truth/quarter-turn-outlines.txt create mode 100644 benchmark/eval/vector_routing/truth/repeated-multipage.txt create mode 100644 benchmark/eval/vector_routing/truth/rule-through-outlines.txt create mode 100644 benchmark/eval/vector_routing/truth/segment-cap.txt create mode 100644 benchmark/eval/vector_routing/truth/shape-cap.txt create mode 100644 benchmark/eval/vector_routing/truth/sheared-form.txt create mode 100644 benchmark/eval/vector_routing/truth/single-logo.txt create mode 100644 benchmark/eval/vector_routing/truth/stroked-letterforms.txt create mode 100644 benchmark/eval/vector_routing/truth/thin-filled-rule.txt create mode 100644 benchmark/eval/vector_routing/truth/thin-stroked-rule.txt create mode 100644 benchmark/eval/vector_routing/truth/transformed-form.txt create mode 100644 integrations/javascript/coordinates-integration.test.mjs create mode 100644 integrations/javascript/coordinates.d.mts create mode 100644 integrations/javascript/coordinates.mjs create mode 100644 integrations/javascript/coordinates.test.mjs create mode 100644 integrations/javascript/fixtures/quarter-scan.jpg create mode 100644 integrations/javascript/vector-contract.test.mjs create mode 100644 integrations/javascript/vector-routing.test.mjs create mode 100644 src/page_geometry.zig create mode 100644 src/vector_evidence.zig create mode 100644 src/vector_routing.zig diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 30488fb..00b7983 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -145,7 +145,7 @@ jobs: sudo apt-get install -y tesseract-ocr tesseract-ocr-eng poppler-utils tesseract --version pdftoppm -v - node --test integrations/javascript/*.test.mjs + node --test --test-concurrency=1 integrations/javascript/*.test.mjs python: name: Python ${{ matrix.python-version }} bindings and wheel diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..d3947d9 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,34 @@ +# Changelog + +## Unreleased: 0.4.0-dev + +### Added + +- Schema 0.18 page geometry and pixel-to-PDF transforms preserve raw PDF + coordinates through quarter-turn OCR, CropBox and UserUnit handling. +- Schema 0.19 page and region route evidence explains bounded vector candidates + and their regional OCR requests. Public records retain counters, limits and + exact candidate bounds, including candidates sharing a bitmap crop. +- JavaScript adapters validate the new contracts and expose coordinate mapping + helpers while preserving text, chunk origins and source locators. +- Small reproducible outlined-text fixtures and negative controls cover native + clipping, transformed Forms, rules, highlights, logos and resource refusal. + +### Fixed + +- Rotated OCR preserves recognized word order. Inferred tables on rotated pages + remain withheld until their topology is qualified. +- Partially clipped native spans no longer suppress outlines outside their clip. +- Exhausted native coverage accounting abstains instead of publishing zero + coverage as complete evidence. +- OCR text elsewhere in a reused bitmap crop cannot mark a disjoint vector + candidate recovered. Recovery remains text availability, with completeness + unverified. + +### Changed + +- CI runs adapter test files serially to avoid process-start deadline contention + observed during concurrent local verification. +- Vector collection and classification retain explicit work bounds. These + bounds exclude initial PDF decompression and do not establish a whole-process + memory limit. diff --git a/README.md b/README.md index 30b5cb3..d54eb41 100644 --- a/README.md +++ b/README.md @@ -548,8 +548,9 @@ The minimal specialist config shape is a JSON object with optional `ocr`, `enabled`, `executable`, `args`, `rasterizer_executable`, `crop_dpi`, `crop_padding_points`, `crop_grayscale`, and `timeout_ms`. Formula invocation is opt-in and model-neutral; no recognizer package is bundled. Memory-opened -documents and rotated pages produce explicit crop-stage outcomes rather than -silently invoking a text-only fallback. +documents, invalid geometry and unsupported rotations produce explicit +crop-stage outcomes. The development schema `0.18` maps declared quarter turns +through shared crop transforms; scan orientation detection is separate. The document manifest is the top-level intelligence summary: input SHA256, parser/schema versions, page count, encrypted/corrupt flags, route counts, @@ -669,7 +670,13 @@ and ruling lines, invokes OCR only for scanned routes, then reconciles native, OCR, table, formula, and form spans with typed provenance. The versioned JSON, artifact JSONL, and streaming JSONL schema is currently -`0.17.0`, with parser build identity `0.4.0-dev` after the `v0.3.0` release. +`0.19.0` in local development, with parser build identity `0.4.0-dev` after the +`v0.3.0` release. Schema `0.17.0` landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6) +at `9cac608`, with all six CI checks green. Schema `0.18` is locally checked and +uncommitted; remote CI and merge remain separate. The `0.19` vector-routing +continuation is locally checked and uncommitted. Its 23 outlined-text and +negative-control fixtures, 13 real CLI/SDK tests and both JSONL readers are +qualified in the [implementation evidence](docs/parser-host-implementation-plan.md#bounded-vector-routing-019-locally-checked). Every emitted record carries a `provenance` envelope with document and source identity, input hash context, artifact id, page/bbox, source kind, confidence, related span/block/chunk ids, route trace ids, and route reasons. @@ -692,6 +699,36 @@ Schema `0.17.0` adds content-stream and Form invocation scope to spans. ActualText and classified text boundaries survive adaptive reconstruction; Form text uses scoped resources and transformed geometry. Native and adaptive text quality gates run against the same born-digital and hard-font floors. +Schema `0.18.0` adds nullable page geometry and OCR/formula pixel transforms for +declared quarter-turn pages. Source bboxes remain raw, unrotated PDF user-space +coordinates; UserUnit determines their physical size. Display geometry uses the +effective CropBox in physical points, while rasterization uses MediaBox and +retains bounded crop budgets. Legacy PageInfo dimensions and the manifest's +`display_transform:null` remain compatible. These mappings do not detect scan +orientation or establish visual completeness. +Rotated OCR retains recognized word order; inferred tables on those pages are +not promoted to table artifacts until their topology is qualified. +Schema `0.19.0` adds nullable vector evidence to page and region routes. Bounded +painted-path collection and geometric grouping can request regional OCR for +unexplained vector contours. Rules, simple boxes, native-accounted shapes and +isolated symbols can be withheld. The collector caps retained shapes at 512, +segments per path at 512, operators at 100,000, cumulative vector-lexer input at +8 MiB, graphics stack depth at 32, Form invocations at 256 and candidate regions +at 64. Page summaries retain operator and byte counters. These lexer bounds +exclude initial PDF decompression and do not establish whole-process memory +limits. Native accounting uses rectangle unions capped at 64 intersections per +coverage query and 1,000,000 native-span checks during page classification; +candidate-merge coverage has its own bounded check budget. Exhaustion produces +partial evidence and abstention. +Partial or limited evidence remains explicit. Grouping uses known display axes; +candidate bboxes stay in raw PDF coordinates. Exact candidate bounds can be +retained within a larger image OCR route to avoid duplicate crops. These +candidates require retained OCR text intersecting the exact candidate extent +before recovery is reported; recognized text elsewhere in the bitmap crop does +not qualify. Recovery and merged route bounds leave completeness unverified. +Vector candidates do not establish recognized glyphs, final visibility or +complete coverage, and use the existing +OCR engine and crop budgets. Host adoption remains explicit; accepted schema families and parser/compiler pins must be updated together. diff --git a/benchmark/eval/vector_routing/.gitignore b/benchmark/eval/vector_routing/.gitignore new file mode 100644 index 0000000..34aa093 --- /dev/null +++ b/benchmark/eval/vector_routing/.gitignore @@ -0,0 +1,4 @@ +# Small generated correctness fixtures are intentionally versioned here. +!*.pdf +!font-license.txt +!truth/*.txt diff --git a/benchmark/eval/vector_routing/border.pdf b/benchmark/eval/vector_routing/border.pdf new file mode 100644 index 0000000000000000000000000000000000000000..7494465acf43bdaaf22defd68c7dc95d52617347 GIT binary patch literal 678 zcmZWnU27XL5Pa9Km={C)&`3J_+!#WjPHhVPNQe(w2z`+6)~<{#6`ky+`SsmB+i_A0 z4sKVonwj10-F=+Cvls7`g+ZLJ)YTPq_4hYH-?;47aS46rmeitPz&)<76@^BC62C6k zw;{v6DD0mVP9sC~Y&*{#vCv!Sv1@66^NH3w%D#5clTgRT2Uhjn zhYz9k&xw;Vp*~k8J`yp}-iZE6jd$bthzG;?L1EB0h6UeK8@H)u$I^;UK=)%I!leVy zKW^tMsK35hn&hxXzvsud;`Pw$Vwhirn4SpE`WCK3)LhoLgtp$?a7A1F(&=ApigX literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/chart-labels.pdf b/benchmark/eval/vector_routing/chart-labels.pdf new file mode 100644 index 0000000000000000000000000000000000000000..d751d387e2cc57a0730be54628980798f153c9f6 GIT binary patch literal 7878 zcmZ{pOK%;=5ry~nSIovhvdFNz=M5MJB1J-gB(Px#NB|=Xin@|0NTfkhiIZRN?^JhB z*GQyRyhlAf^*r^M?|%Br%Rj|_ImIq?w{Je4zW=_vc>TX$&)voI%ZJNPx9_`)pDy2@ z@4Li-pSvG^IGu0a>Y#ofAIRq}{&xNLzWZg=@qcs6?U$Q}F1iPYZytPk%aMy$U*0^} zi}m{0T|B$IKj;7X>-p1v&JWjDm)*sSo2%Qm*EdM@&+D5%-`rpCK$A?b{#G*Sad+lL zS&@r>oZntwKD+&``z3IGOraY>K;WOx_qSi}uFzPfFahk3*hz3%pu70jk8eJnuO4(l zzXd%<2X4RbrgEyHgvFeEInly7#ANuX$PQV(MPM!(5Un zc4O$$6uQ@+yFWbtaC!I8U0weAaD98z|FL`hv3v0vul-ktIPDn*LFb>GtghS+Y>(^o0jyRWieCnq# zY+;sgYwS^0lE&#NJoG~vQ{3$Q){jd(h^Tl=t@fpt1f(>iTk+|V&@5>zq=a{+Py_1m<_9dSzi8pAX$oSynostApkJZ-b? zXei!V23)a~JL>6i-XC-~k0UujA|2%zhki8U*&R)4(j&!r*(68P5P#1I#~Zyp8Y-JM z>owe48WfWg&j-KoKKxFM>x$+?V0oNS@D4|4Wgn+~cz~M8likt6i5Cu|a=nHPHc;}7 z0YN5_guQVXwv_+@9cJ^hg^hX9(BjV=Hseb05vl{Z9E=I1c*$9zg??nFnv4yV%U6W8 zjCH5uHsjINf($vvB@h(m@}z{2bg2(plv{KM`LKx^z#*_m&=MNi5gD3Z7;5#-TB z*tF9G*;Iq9-8_jJw`Jxf1XGoRpoXax)RLn~O#>e3UqF(T&d$JJ68KPo;QfQ|PeaFFaN+*Lpopfsd@ zth54=Dd#3p>OkO^C=-=sJf*E4L?>jtQC61{Rl+zAWatSg^ub*T%bN*+Np_)_xreY# z-_UQP1*HQ$Q+aqMyD5>7uDcYXs(I-oAi|3^Btnx_L}gy?yk6w|l=;_Np`qL{jD57C zPxiRe76^I>r!lrZqW+1)L&7TsWx*XtWl{rlCh&1}2A7uB+|b&AX9@CWZIM z&_phdbE-K9KN=1Sd~c$fmI{-x-_W@h#*t67E3pz!Lu_K8f z92yBRF0ek#R(S@>Daqu~L$tWlBXpJ<8-KH?ZxKBb08I{@7cG@djwZEeHFLMuuTitN zMMHDQZMna6gsl__3A!a|>}E@4Mffz$MpGt-8BBLfbOoi%AoUy6tN$Vp)B~kyP|+G;ZH1cEZt%+ev@#s1r?<7##;Nx1bXI$PwG5OLSX#o0jWVf9 z`P}YS03kCy^U#=Yg?y!xgwTWZgL#5Tn0d2yAC%5YEGKM?xtRiMZ3h{=swQWZidC3? z2*%t|qG;_^$E{bPvhuy7$j*he(`=`a&f``qkR8yHG3)674A9CHZ<i+#Q)-K1aL6Nz zM-P^Dr)l(ar*XI$vzdjCa>&R=sj_U7S}svZkkn1XW+`uvhH{6qU&G6o%&cBq&UjQ} zmMNK$e5BE3)M&6q$Ohw}eMX*OhJnnCrM@x@jF$%re?eRadN^B{a7|ly zpo!*rsO)fhBmG3N=?Je_H)mQF)=N~p$Sh)n+O9)X1_RBfvh(g*bZv0Hu0DJGDUgrHpB7qA#F2HUVTG7G1nk7wp0k z=^>9unhx?C6#bdC+`QaCo^R%AuzaXV)he(L38;@@Tq_c)ly^u9Tb+~|ERo9^O(S7p z3N&VbSERX_QFBh2Ql(uUbW^0Rmn`O{tdRo=T4%`p?8;x1WV_MUC32Xhh z+vY_HYO&O<@R*HujEksb%?6;Y1d87LlHGRmSeUxs%>ZSU6QeH`}GLQDG*?FhL1DQOZm^0htyo?<;lnakAK>HZrMIc%D=X7U6*x<$X3)d#l@wT)ne))IZx$OU zq4r%S1*9}Uz;%^5!{>#7+a{BhxizE2$_^6X6J5OR0vwze4{Ap5YfN=_77twDwwP+c zEMS56C8ogA*u!^ahv`72bX{S3c;8^UTSd5ef$1b5*Z2LUla@#+>r1B>;aj(t%?1s` z2F+d#PR+7_b#>XiAY!qKqUU#qf!4)kb1lNwcDjWDV4EKt?6;O&v1$>$8DEv9rCt_n zGY5&)oh5RW5fiYjEYTt64svN5OSDzUQNaM~w4)CyIanL0=wQ>~j}Zgbkta!17L{sF z%Qgx1_dTWh&Dw)7_2xCD+RB>M`{|MODlJZbq@&nhcq{2aK69|grDg$!! zdn1c7XwTYp*#?12_Wn4@?791IFofk`UpAJ#;9-|mO&sjYhC(N=I}g>Q!H2TZE_n*} z)r3ptJP(phdw(psD{R%IX`P9-{r2jLX!(*`zCf05y1(6>-|;naK85{XH&1i~-Mg+~ zTW9~<-85`kzrkYU%cuUibM%1UV8g`s+IsHc?(+K6`OdyVzq`09&o$=& literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/clipped-out.pdf b/benchmark/eval/vector_routing/clipped-out.pdf new file mode 100644 index 0000000000000000000000000000000000000000..0daed1f39cd5d1aad123b69d258a8e6e94a93db5 GIT binary patch literal 7956 zcmZ{p*=`)i6^8HUDe6W*a*<~33t$+C90>uk*s%pTfRPJ|8p#y6B;-hN^7Q#mRZrJx zQ8WaD^G|j4S^u-liy!{<`VVd?2WPCgfBW(9%{S)a&42%WG#B4po-S|hKbVUjESLu?Ww<5 zUhmAs%ge{(_~y^Yo6pD7_0^@hcy)Jm|L*z@seZn``{Ui?^#)AI&ut-znNbw=cBhKSc||v9v|;NJzSx&n4$~N|8PbG7Ymq+e}4b={~nI!XLEOm zb+OJB*Nn^7+#aG`ysIS{SM3s=cea_ELkx?vA-Vho+?G^qKq8mJ;(bopb2{b4CR-}9 z>Pu2x+E?yQZgJ74Qo97aYbVjo;n&0aLv~BEKE$YRygih-R9jQq!8aN%TvkhM4zY4v za*h@e-E**CY>dr|65JhXEE$!{*Wuij;w}2`?|u&UOYqTSXzorJDCR0EKLcC2bZU)n zcAvFf#Xt>9pr#dR>rz9j*y?r%C#+veh+Ndd(oj>JUh+~}z$VQvEmdogsgxx~TT(Jj zBuKu54%XaUbmr>N>{63YL7uORZzcI`D&`2Gc$@*4Mo?A$inV&Vbpx*sH4|kiHo|Pe zH7?ON8$^eKR5fARO61iRlSq&kuD1mNDsshE3t;50ltVROf&7#>!eO}yW zpFf|?DBE(0NzSg9=Cs7n98#;-6F%1YjLf`$ii!hM%|0XQR0_}45-jgh2ACihOk+)0 znU!0KTWqO3Aa8PHmA6#Gs>%EV`|RDY*b*NKukz*RjiikHYTE}>v9lGUi1MhXg;2aRBsb9PM- zvm~OaC!juy0NDkJAznt*5IN#3+ru2tOSnc6ptL1ek{sqi+YnqKBZ#o*O3OlV8LdKe zPJ=}ShB{lq5_2xC7huYS^;hk&Jj}8L-v}2Z+19WZDi{#_k`3)OiA&t;+@o}D@)4IVUHzZ zPy_l&CD=3Yly_7O=i`z>K=A9Er!r!mCP17l+41ZI9Wte>P=SGwwS2g;pnG&gQ&<_ z;xs_~4r}zAM`n`&0oWH5Z9>`lC^jI%5mFNnT8J7d0B6 z)>AhtODj3FTa?h#g-So=?3qFvIEd3l{JeUR9&l}p2^t#Ol$yuza5Kh%fYC9w#RV8d z@`%P5(@66Mnj7AbT1977rQnO55^ecf1%xR$Ag4jA82JdprkM?y1kQ1+T?&3KgvXor-So8{!7NEanE#QKBB2U^qBudUwW061OXS^#C z@z}ZD5Zt=ZD5j!Uq{*W$20vc$wKX#c=zMzbb595F;YpzV}-Vhb70xLLP$Jn6%=KJ1v;@;XWGa7LLi zRqb{du0gXDGMSO}ac{{p*=-P^8h7;GMfiHifkXC!4X@IQl7b9SB^;(>rGtz>K*G* z>1cL!MC^@j4$;4e6$p?`M#^ay1tPjHwHHQ7=GqwrE0B-tW`{a*9yC3@8jS?zBUPpU z?NhO*`)%iJ5BRBX{)N~;kz*ZDaQoK5b*WQg2`BkS z_Z!*DbOoiik>-Y-qp|}Z#gx-vBbpWPB%0}L&qMwE4(n1jyqIQZMWR{Z_-SK88k$Ai zE|NZrE7GH3V+8Lbcm@3H=;CghZzuA+j%Aa_JjL8Htz_ zF{`Mj@bRQl;R&GM*}1M#lvm!xtJ~(225{nP_333x*Vc{XdH2clSXcxiG`8g@Jrkt7 zj{Y#@dHd;E3VI72z+Sv^U4Jq`5o;un^ec2DWB*BuytP*oB?IIG1Yo5Tny1O+a;AW7 z1?rh!&J!yK&O1;K>yh@o2^9hyTTmDnD>D>|aN8PmhpJgp<3hw^5vp$%(jKc&Ee5=> z?Lt`>ZbIYpbs4G^l%2xZhH7(66mux!OzeoOX&=z!wZwWz$$2ReMIVKmlbFDlY84NEHsxGhFS6AN2QU&{==CLC=5 zla)+lbFLN(v84Y3m4J~6nzbBkr%n?ZGJ#9?&imHgsCboqWdB_RQ=FHho;F(QflPqg za@134xBTC>qZ;MFwm0A5ykv~K>SAJH>_@c>h(7C3=fVR@K!+U8`1{hSW0Ij)D^*`kI&d2e4=K2fqZx6@!{I;17tgF`lOiuJ{=DnF% zGyOYrH!-sAON{R12}XbL1PlCnuswTvxV*kOKJ?$Tf4ctP(PVz!y}v*8-&gx@*^j5o Shf|l6+T{-~UcCDL_2K`PKS<;N literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/compound-fill.pdf b/benchmark/eval/vector_routing/compound-fill.pdf new file mode 100644 index 0000000000000000000000000000000000000000..a3791b94b294c1b1873ad8d3cc533ed28710a28a GIT binary patch literal 7907 zcmZ{p$!;CT6^8fsDS9IyS)^I>02l@$M?!!MHf#b3U}Qm&R}uvd0eK}jd3t}Rs=Mp7 zsOyCx{;aM#<9`Oe`spuk|77QUu#=e{-+elK_uX`H`M>`jr;8u1PFHu2AEt|+u09-} zC&z)Gr|-W%93O7vp!~f5pr5<=+wINs^lOmg|KXO$&kv``st^5(58pm=jl?@sMu zdEQPJudkku{WpI--hDZqZm+MVi#HF~k2kjuNcFGVhd)0&-)_K=OrCxzndtF!~M`*gfM$qD&c zpdZkI+n=YnoLcQ9Kd;jK{rK?V^l{4GCi(F2e0n-w-5<>K;qdyB$LwU?bom~0iA;71 zW^&O?m-o{j{&{t}{c@a+Uv6)X57)=(>3Dy2`|v#fak~68y}3MuG8=2Nt&>e=x<3Rn zJ6m#`Y%z1N&YC*i9fF^&@sa2EaGhf|p2uzSvvVmX$LW}66HP9Nr7wwPuHU#j+Svvd zb7>Or)=r|k!^cCib2ZL~pzpgsO=+Z z_9f(m!ui{9s&jS*?e}+|gWc?1aG069BjyRIh{9)JBbN@X@yzaLZPky5iXBkWh^%EU zz7{NXzk?%|&)EmAseY~~DGtwR&eda)>gF1YF-Vm29D>O)PF3EIF8cIk$UuIWxbZIIW!~5)T1z_v}RrpsZ#mTK3czq~|2#cW-VgoK=4z3z6I%K3O5yKXu zuGA1kf;4lz$p}!8E7Xz!1OJN2mkG?1h7v=#s}vtn*nx(dVhqdo>)C{|HRTZH>~d*} zbMVz7wR)c5G1ety=KUcm?u#Y4gs4L){8q9zyrm2{rK)^C;)VBPCU4Uc7}jMxkjz~9AKA`RrPLH8RG|as z#hZv#OpQ1IO#XqZT3lo&ku}+ff+m118Ty<2S@j@5Z&}=2gY-s7mYYX?^ zAFL&yii74g{od_xF4~CJ8t(D2l9rO--^3fCQBmfYpu!eOM4$%rmrAfR@Q_#Pg>_+$ z-Xn680Im|+M-xXDxm;K3>9}=-yN}MHU*jGk1&J&{EcDgM$JHc7@fxk6G8jXOQ7b@> zP~x(ZmvT1P=8CWA%`baQZ;)wZ>2pwZy51^extCH}||I1IUFS zWF>Oskmx75!(s%Fl!z*1T}sLl>vTrF95uEUW|O=a=q_qBKCGv1R+d(BXtyY#hYOW{ z%Go1@Hn2rZz{`sd>HU^=n2@23jj=k+4l83Cs22@GovnvIB#UT=F^sfrpsC^usZ#V* zMM}L`DbkED{4YGg0XYp?EN@EI(!uqv+tNm_RaMxd7ZAB4(E123buu`-ASXU) z%*A)kCn>gYvFPOk#lO9>lx{&CaVL!)5=CpMuE?M8GQJgw_-t)71UJ@K%BbiSXz%Ds zY=|>D&|@`}_NhvT7a+H|ly5}pDDnrGWa%QgW5Go}g-d&Ey;v^Gys~Jf09>}sGV)S% z_y-;Kc%riwEC;-Y4lszcEi}~KW2N?CM&c~P8eE6bmv%xL z*=TIjSfg8X$sOu6Q&lL^OA3k+p&MRwxRku%abh867#oqY^uW^miyX*D1#*ps(7iOT zw&?3E1|ccQB{lk+0o+4-m>4COAx%PC;lyI6J8{V{EQzY{C?f+VN5~DA;3!0j1Q?<_ zBFRXC5LKRzPe_#lW^3=Q#0^Xhq!Ktp;;C#LmK^Jo6#R+u9N(kbuKIK&Z*r4JUrZQ- zHvcLy2>E8%W>+P2mgb3~z>7-3%}_gYZ0J6shgntrO7%`u z3eY-Jqh+Y5nO#Fmbj-s%K#sJ5S|7DcYH67tcQ^w~pQUzN3tORBN|;Q>`nWUXm~1nM zPK{eS?krs0<-i?#!NjAqpd=qd6RA4By$7dt!QcbC+pwm-{_q8$zv2x`kT3F$c;XF( zWBW#^vIue1!lhN+!8DYKp|6>BJML&GsmL-*NCBtMi*$gFhSGS`b_0}e%d7)0T_2o_ zWFei9n1nex*5o6nDfY7YAyX(1p_8IUVTEOR)bChN6OUDjQ)zdsH_O=1&`*=D<&b93 zNBa0C$B-Htj;75})V)o^VJKye0V0O7-oTqrEqhPtV|MjJ?2T>~(Z7llh>zVxOmPKIwNN$%0}2KF#bLFs9vsbTY|?7)v=%3-h` z%L@1^mg#2KA%6S@+fpo?7-nZfVp-vMX=6azm|2`Ia-PK%+0mvkaMc9AfT+L!=S<49T zEhG9K-=9b3Kv0p)=kP8f8tsUX?ckhp+c(M(#B7LBMLC7{FO>=}0R1K`S17{^+v3q} z%elkVn$yWft}ffhbkbwcI9y6<%G9SPssPNBZO@q~M^ zF+-pTx2-&PsG1!$E)?9Cp895?>%R8XV!#O7-jmhfCNv&j7N1%{*&+1Jr#8nxF^4kV zz_p>dnJi|$&|Z<}*6pYEA$VVZYHyGIr;ZIY*%qKXTx~$xEy;PJ19Dqdpz@yc^z>4_ zXr=oORHS42mF5p{TY`!v2A-BqmYI1;H_AV=l+0jbjus2CkpBvmfRTxrB^*qrP7@k3 zaZA6>d&?eFyvpve{RIS5tcy@b<1CdxX1;9^>Zq|>_HUa|jdEbyk?(LG=>x91m{{n$ zP%Q(ZkBX%vroJuPP_3O9psho-DK=1I1*=8Y)4d;C&&pq-#6XRdmX&DpMoFyE8oKR7 zo9_}`4emgQWhvU+Tuj`rMRib+scJ9k=oi}#V^@6c5A675-_izIh|vV)c!>M%kBSF3kc)2Yds*_$6;y?XQG+r#fH(mqrG literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/dense-graphics.pdf b/benchmark/eval/vector_routing/dense-graphics.pdf new file mode 100644 index 0000000000000000000000000000000000000000..7778f5d37db29c0de0a870f6ee0b6dbac850bbb6 GIT binary patch literal 1636 zcmZXVO>Y`85Qgvm6>~vq56#-UEFn@9sidGv{YaFo)Jjng1`IWV-D=mMw7C2dlvRKKILhvAuK7glMubs&>878i|^1|`|^b{gWwyuG1pWIMNi^VzL;F8GX`d=*h3 z{xQVjd;^sIe3-7>EN}%gNB1}bR_~}hR2}b$`DpXi)ywdkjP94*sJhTPyHT2!YE)q} zLnfzcf$L%?Lw!wosj1q~yYDvmog=sNb63wCwQgg5-Cez>YDMEpp%j%v{8~o~buH28 zGcL|}aK<;1b6iilJX;AKN-m6K*uf~ZOOfpa!%i^l1j8<6wi66B!LUo2?NWvvR!r?u zWIMsI6AU}SfRx!zFzf`wE@ie$8FmIyswbHJWB@@Kz)%KQUj*zZ14zmMmLgzH5wItA z7M>|bPZ6-E2-s5u?3o79Qv~cO0`^RU=$QuWi7w%pa`Y4dxT(R9^QFk>{~Qf4E}MqO`{CQr1>1(`Y5HZZdcE<(MEwWQDtWg6 literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/filled-box.pdf b/benchmark/eval/vector_routing/filled-box.pdf new file mode 100644 index 0000000000000000000000000000000000000000..5c4e018117557240b57fafcef6df0952e25abd71 GIT binary patch literal 679 zcmZWn%Wm5+5WMp%=3<~d)Ffp~je#IQo!ThSM}kB^0t7wK(psfqlZ8YZX}>kVab_?wh7I3Q7q@N2iDeYMf&RxrjLQU| zA8!^bs**6I7U-HY@aY}$1M4^^sgG{HQE1XHOsur2F16$~%AirxlBFzchH}Zn5=Rze zgDf{FH+c8MCBGxk&es$wLQ5OxLpOYna)nu`%&Mf+-^h`dTIbi)_V4W8|0NVW@)+fb V9Nt%#T$_5I<4K_ggW1hoy#Qojvt9rI literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/font-license.txt b/benchmark/eval/vector_routing/font-license.txt new file mode 100644 index 0000000..6849665 --- /dev/null +++ b/benchmark/eval/vector_routing/font-license.txt @@ -0,0 +1,57 @@ +Bitstream Vera Fonts Copyright + +The fonts have a generous copyright, allowing derivative works (as +long as "Bitstream" or "Vera" are not in the names), and full +redistribution (so long as they are not *sold* by themselves). They +can be be bundled, redistributed and sold with any software. + +The fonts are distributed under the following copyright: + +Copyright +========= + +Copyright (c) 2003 by Bitstream, Inc. All Rights Reserved. Bitstream +Vera is a trademark of Bitstream, Inc. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of the fonts accompanying this license ("Fonts") and associated +documentation files (the "Font Software"), to reproduce and distribute +the Font Software, including without limitation the rights to use, +copy, merge, publish, distribute, and/or sell copies of the Font +Software, and to permit persons to whom the Font Software is furnished +to do so, subject to the following conditions: + +The above copyright and trademark notices and this permission notice +shall be included in all copies of one or more of the Font Software +typefaces. + +The Font Software may be modified, altered, or added to, and in +particular the designs of glyphs or characters in the Fonts may be +modified and additional glyphs or characters may be added to the +Fonts, only if the fonts are renamed to names not containing either +the words "Bitstream" or the word "Vera". + +This License becomes null and void to the extent applicable to Fonts +or Font Software that has been modified and is distributed under the +"Bitstream Vera" names. + +The Font Software may be sold as part of a larger software package but +no copy of one or more of the Font Software typefaces may be sold by +itself. + +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL +BITSTREAM OR THE GNOME FOUNDATION BE LIABLE FOR ANY CLAIM, DAMAGES OR +OTHER LIABILITY, INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, +OR CONSEQUENTIAL DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR +OTHERWISE, ARISING FROM, OUT OF THE USE OR INABILITY TO USE THE FONT +SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE. + +Except as contained in this notice, the names of Gnome, the Gnome +Foundation, and Bitstream Inc., shall not be used in advertising or +otherwise to promote the sale, use or other dealings in this Font +Software without prior written authorization from the Gnome Foundation +or Bitstream Inc., respectively. For further information, contact: +fonts at gnome dot org. diff --git a/benchmark/eval/vector_routing/fused-panel.pdf b/benchmark/eval/vector_routing/fused-panel.pdf new file mode 100644 index 0000000000000000000000000000000000000000..bb45926ee87b7b04d700f50e9fa4d747246a9b0c GIT binary patch literal 7978 zcmZ{p$&MVy5r*&m6nPO~A6%3B0tf<3S%LvCA|wJfAowsGHpwAoF=m>C?5Ecknbnn1 zBwGi8^LJKOtbZ)^&C9?1_$RlNgEQ7VzW#9d_FHrH>VN+|nyc@xPuF*k@66T9>vzXz z<2mrN`R=>J@!^del)v{6jB{6iyM6O)evNYcKiu;8@!@2gdN6$P;E#_Sx%%bf>r;QR zyxy6s7uV0n@ylP2cb|@@+nZ~1^~1x>!D_3xive>mQpfZd}pVz0`Pe*h7bo=J`aC0dEAi*>HJW?Z)B{t)fr zT`kGDYM1D|v(4NcVpyCF$>q=BwxnVMFLyaC-shA(r&C^RvZW%Hz9iM9edX@t78iXg zwM)Reb`sqk-XF4Cn)M+@ec%0|#HHGr+77U3@FaXHzjo2*u+Hz%&A?@~>E{ms>aR=1?;cmSQ8sCS2nZeX~JyC`eTkhOIlo-NYwT77E4z%2qQe2-upUo)Sa*0XKu9xPt z#Lyg4tJf1e*7=Ogynl*{`%=w5BkEKNzts{f?@|VsAQwzyO<0+gTZvk1sXQQWa%7hX zpu7d#aF;X6t7Kwo&c$q0V4y(ksVdLUc;Wr2khgga4Cf0T$aZP`k8I~vwYD53RG|YO z#G8m!ZA&-+O#XqZdR!DQk!4&$!CWMxQ8kSO6fh48!7S(8njmILL{m>ceHH<-3lc-T zjHn@U#96k7IiQztjUqs4OR%Iktb?{8xI#t{VbPVAh2An+h3K3HiwX>NwuB|-Tv{>e zR$|a5UZLMlu;PF~+@ipgm8WV*8Ddmgk+pClE>s5CupSoK$|FLW;~+s%Qm&E}VA(Z< z>gErUjUDtPZmVK{Lc!%G_UG-bcr4JvBG@Lm+LlNTm!_ivjaRz3fo2;y=3-Ab-nMf8 z{lPf`s(NT%)9>95=c0{ho#md88fhsB{#Cpa8Wm+p87l0NLx0zvP~C>gFb75B2x-nsNj}B{$o7V@e%J(Q}QP8Y)yL zp40fWp1N6C+D|!1$mtg)weq6UkkDBpLEB@Zk^=gdEej=Z$`V>~;*Sk>NP>qN9FWtX)$vZrS~|GVbvxSVb5#}g7zIS`h_pV!OM`Pc=11GQ0?3I^ zI&<-y4_S&WTr7HpNb&D)SxdK|p16}n4~deqR9EEBcp2Y{L_BtGGz7OUG|H&x6>0D2 zN?c3}I?!Wvl=i7fhZiBYw3cr~8YuEdm}KoDxnspeK7~tr?1EUX$h@*@rT|>F%`)<) zvl@mPs?4;^3-XGBSJU4(c@C`hL;lysleEXRHO%%=3nGMJ}Qv!G=%Q8 zd$mX3XfX&$NiM0;Uk%_9+S9}+xejR(+6pHY2i=KFreR4`g(n#qFgZeQxCBQfQY63- z-4jVx5`?G6xffyWaI!{s)$KVR* zrrBxXU7BLbHO9JxtEpD*k-k(cyG_6w$YtB0D^zyA`68}nTDhdGO+Ea-jles+XRlIy z5S0S7&eUibDq3OJ&=Vc=Fb|L;eW2Dy9g|vK=f@q+05fK(-PXc2D3%f?ld(STEjcFJ z45Cxxj*hzsR}VRG$6he;C@m<-$IwKo&S&qzXF;;DsetGa_}C^JJ}H|uuX(@;{8WtNZvPM??P06h(*4YuzFDBq4*2VS~4 zc%R5ZIw3I$b98DsBu-Q8W%EO(P#)qSMUBD=>+)#aaXw8vHYrZE-*LWK#(su=nshCP zG=m}0$9FlV)X;FUeTJg$Z5mEPDRWE^F_iTt-hJxWdrBX(s~=)-baROQS)4$8>@HGH zyBHABeW|@5N*dSBBv^q=Tz5ItkMn5h>Ck8(Fds=OU2mU*Jw0zb=Xt}*IZD;zIv3`iTZh|@*Ri?|{?+B8P)J|^x+P~)2K z^!*oc#djiST-#afx$)o*7akDPUTp$04=9wb2~RU$7M&Um;S2{U^@VEQZ## zjPTw%qVMtjd1Q_R70G-FA0ndBjtJQfF1hr5qYOdJh8R_pQ}}pNsqh7$-wDfA%J9mz zcy!xx?r^o{^s-O=y=j3^;E8-9q+oF>$2{K+seYo+w>GUiBy@d$iEnc~V= zGg3!76#9>`>!c~(8mfsB`|<~=Z>9H{r^(@RPX2A(>6ur~6DkMJ`%Vwvk?_6k6zUrr zPq-HwGX#oo+sbo?s@YNFLcwF{sc#m#9&1l628^)nJy{KILgVpu@u?M*9m3drYI95! zb136YTpOC3$ztXU?G<_5dHbn-2tL-I+S_ygsbd38wgu=8R~yiNOZI{2fZWyq@kHqh{7< z9o=@K-FKO;26rIDx)kkht~Tx0qB=B=4nGyOO7FfkJMbBxyJ3yk*S3oP)lVSD!UbbWhweCl7if4Tkt(PTdQ fK0coMSJ(c9`}uVJbn0>zhMvPW-~8~?kB8p@S7=Ux literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/generate_fixtures.py b/benchmark/eval/vector_routing/generate_fixtures.py new file mode 100644 index 0000000..6ebdfc5 --- /dev/null +++ b/benchmark/eval/vector_routing/generate_fixtures.py @@ -0,0 +1,291 @@ +#!/usr/bin/env python3 +"""Generate small path-only text controls without embedding a font program. + +The default font is Bitstream Vera from the installed ReportLab distribution. +Only its simple TrueType glyph outlines become drawing operators in these PDFs. +An explicit --font permits regeneration without ReportLab. No packages are fetched. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import struct +from pathlib import Path + + +def number(value: float) -> str: + return f"{value:.6f}".rstrip("0").rstrip(".") or "0" + + +class SimpleTrueType: + def __init__(self, path: Path): + self.data = path.read_bytes() + count = self.u16(4) + self.tables = {} + for index in range(count): + tag, _, offset, length = struct.unpack_from(">4sIII", self.data, 12 + index * 16) + self.tables[tag.decode("ascii")] = (offset, length) + head = self.tables["head"][0] + self.units = self.u16(head + 18) + long_offsets = self.i16(head + 50) == 1 + glyph_count = self.u16(self.tables["maxp"][0] + 4) + loca = self.tables["loca"][0] + self.offsets = [ + struct.unpack_from(">I", self.data, loca + index * 4)[0] + if long_offsets else self.u16(loca + index * 2) * 2 + for index in range(glyph_count + 1) + ] + self.metrics_count = self.u16(self.tables["hhea"][0] + 34) + cmap = self.tables["cmap"][0] + formats = [] + for index in range(self.u16(cmap + 2)): + platform, encoding, offset = struct.unpack_from(">HHI", self.data, cmap + 4 + index * 8) + subtable = cmap + offset + if self.u16(subtable) == 4 and platform in (0, 3): + formats.append((platform == 3 and encoding == 1, subtable)) + if not formats: + raise ValueError("This fixture generator requires a Unicode format-4 cmap") + self.cmap = max(formats)[1] + + def u16(self, offset): + return struct.unpack_from(">H", self.data, offset)[0] + + def i16(self, offset): + return struct.unpack_from(">h", self.data, offset)[0] + + def glyph(self, character): + code = ord(character) + count = self.u16(self.cmap + 6) // 2 + ends = self.cmap + 14 + starts = ends + count * 2 + 2 + deltas = starts + count * 2 + ranges = deltas + count * 2 + for index in range(count): + if self.u16(starts + index * 2) <= code <= self.u16(ends + index * 2): + delta = self.i16(deltas + index * 2) + distance = self.u16(ranges + index * 2) + if distance == 0: + return (code + delta) & 0xFFFF + value = self.u16(ranges + index * 2 + distance + 2 * (code - self.u16(starts + index * 2))) + return ((value + delta) & 0xFFFF) if value else 0 + raise ValueError(f"Missing glyph {character!r}") + + def advance(self, glyph): + return self.u16(self.tables["hmtx"][0] + min(glyph, self.metrics_count - 1) * 4) + + def contours(self, glyph): + start, end = self.offsets[glyph:glyph + 2] + if start == end: + return [] + offset = self.tables["glyf"][0] + start + contour_count = self.i16(offset) + if contour_count < 0: + raise ValueError("Composite glyphs are deliberately outside these ASCII fixtures") + endpoints = [self.u16(offset + 10 + index * 2) for index in range(contour_count)] + position = offset + 10 + contour_count * 2 + instruction_count = self.u16(position) + position += 2 + instruction_count + count = endpoints[-1] + 1 + flags = [] + while len(flags) < count: + flag = self.data[position] + position += 1 + flags.append(flag) + if flag & 8: + flags.extend([flag] * self.data[position]) + position += 1 + axes = [] + for short_bit, same_bit in ((2, 16), (4, 32)): + values, value = [], 0 + for flag in flags: + if flag & short_bit: + delta = self.data[position] + position += 1 + value += delta if flag & same_bit else -delta + elif not flag & same_bit: + value += self.i16(position) + position += 2 + values.append(value) + axes.append(values) + contours, begin = [], 0 + for end in endpoints: + contours.append([(axes[0][index], axes[1][index], bool(flags[index] & 1)) + for index in range(begin, end + 1)]) + begin = end + 1 + return contours + + def outline(self, text, x, y, size, compound=False, stroke=False): + scale, cursor = size / self.units, x + painted = [] + all_paths = [] + bounds = [float("inf"), float("inf"), float("-inf"), float("-inf")] + for character in text: + glyph = self.glyph(character) + paths = [] + for contour in self.contours(glyph): + points = [(cursor + px * scale, y + py * scale, on) for px, py, on in contour] + for px, py, _ in points: + bounds = [min(bounds[0], px), min(bounds[1], py), max(bounds[2], px), max(bounds[3], py)] + first, last = points[0], points[-1] + if first[2]: + start, pending = first, points[1:] + [first] + elif last[2]: + start, pending = last, points[:-1] + [last] + else: + start = ((first[0] + last[0]) / 2, (first[1] + last[1]) / 2, True) + pending = points + [start] + paths.append(f"{number(start[0])} {number(start[1])} m") + previous, index = start, 0 + while index < len(pending): + current = pending[index] + if current[2]: + paths.append(f"{number(current[0])} {number(current[1])} l") + previous, index = current, index + 1 + else: + following = pending[index + 1] + end = following if following[2] else ( + (current[0] + following[0]) / 2, (current[1] + following[1]) / 2, True) + controls = (previous[0] + 2 / 3 * (current[0] - previous[0]), + previous[1] + 2 / 3 * (current[1] - previous[1]), + end[0] + 2 / 3 * (current[0] - end[0]), + end[1] + 2 / 3 * (current[1] - end[1]), end[0], end[1]) + paths.append(" ".join(map(number, controls)) + " c") + previous, index = end, index + (2 if following[2] else 1) + paths.append("h") + if paths: + all_paths.extend(paths) + painted.append("\n".join(paths) + ("\nS" if stroke else "\nf")) + cursor += self.advance(glyph) * scale + operators = "\n".join(all_paths) if compound else "\n".join(painted) + return operators, [round(value, 6) for value in bounds] + + +def native(text, x=40, y=205, size=12, color="0 g"): + escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)") + return f"{color}\nBT /F1 {size} Tf 1 0 0 1 {x} {y} Tm ({escaped}) Tj ET" + + +def stream(content, extra=""): + data = content.encode("ascii") if isinstance(content, str) else content + return f"<< /Length {len(data)} {extra} >>\nstream\n".encode() + data + b"\nendstream" + + +def pdf(pages, form=None, image=None, rotation=0): + objects = [b"<< /Type /Catalog /Pages 2 0 R >>", None, + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"] + extra_resources = "" + if form is not None: + objects.append(stream(form, "/Type /XObject /Subtype /Form /BBox [0 0 350 120] /Resources << >>")) + extra_resources += f"/Fm {len(objects)} 0 R " + if image is not None: + objects.append(stream(image, "/Type /XObject /Subtype /Image /Width 20 /Height 8 /ColorSpace /DeviceRGB /BitsPerComponent 8")) + extra_resources += f"/Im {len(objects)} 0 R " + kids = [] + for content in pages: + page_id = len(objects) + 1 + kids.append(f"{page_id} 0 R") + resources = f"/Font << /F1 3 0 R >> /XObject << {extra_resources} >>" + rotate = f"/Rotate {rotation} " if rotation else "" + objects.append(f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 420 300] {rotate}/Resources << {resources} >> /Contents {page_id + 1} 0 R >>".encode()) + objects.append(stream(content)) + objects[1] = f"<< /Type /Pages /Kids [{' '.join(kids)}] /Count {len(pages)} >>".encode() + data, offsets = bytearray(b"%PDF-1.7\n"), [] + for index, value in enumerate(objects, 1): + offsets.append(len(data)) + data.extend(f"{index} 0 obj\n".encode() + value + b"\nendobj\n") + xref = len(data) + data.extend(f"xref\n0 {len(objects) + 1}\n0000000000 65535 f \n".encode()) + data.extend("".join(f"{offset:010} 00000 n \n" for offset in offsets).encode()) + data.extend(f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode()) + return bytes(data) + + +def generate(font_path, output): + face = SimpleTrueType(font_path) + fixtures = [] + output.mkdir(parents=True, exist_ok=True) + truth = output / "truth" + truth.mkdir(exist_ok=True) + + def add(name, pages, outlined=(), native_text=(), expectation="candidate", boxes=(), truth_text=None, **resources): + data = pdf(pages, **resources) + if len(data) > 100_000: + raise ValueError(f"{name} exceeds the small-fixture limit") + (output / f"{name}.pdf").write_bytes(data) + (truth / f"{name}.txt").write_text("\n".join(truth_text if truth_text is not None else [*native_text, *outlined]) + "\n") + fixtures.append(dict(id=name, file=f"{name}.pdf", bytes=len(data), sha256=hashlib.sha256(data).hexdigest(), + native_text=list(native_text), outlined_text=list(outlined), + routing_expectation=expectation, outlined_bboxes=list(boxes), pages=len(pages), rotation=resources.get("rotation", 0))) + + words, bounds = face.outline("VECTOR EVIDENCE", 40, 160, 28) + cover = native("Native evidence remains.", y=250) + add("outlined-text", [words], ["VECTOR EVIDENCE"], boxes=[bounds]) + add("mixed-native-outlines", [cover + "\n" + words], ["VECTOR EVIDENCE"], ["Native evidence remains."], boxes=[bounds]) + clipped_native_text = "A" * 40 + partial_native = "q 40 155 1 30 re W n\n" + native(clipped_native_text, 40, 160, 40) + "\nQ\n" + words + add("partial-native-clip", [partial_native], ["VECTOR EVIDENCE"], [clipped_native_text], boxes=[bounds]) + chart, chart_bounds = face.outline("VECTOR", 160, 140, 24) + label, label_bounds = face.outline("EVIDENCE", 160, 80, 24) + add("chart-labels", [native("Chart caption.", y=250) + "\n1 w 40 50 m 380 50 l S 40 50 m 40 210 l S\n" + chart + "\n" + label], + ["VECTOR", "EVIDENCE"], ["Chart caption."], boxes=[chart_bounds, label_bounds]) + form, _ = face.outline("VECTOR EVIDENCE", 0, 40, 28) + add("transformed-form", [cover + "\nq 1.1 0 0 1.1 50 75 cm /Fm Do Q"], ["VECTOR EVIDENCE"], ["Native evidence remains."], form=form) + add("sheared-form", [cover + "\nq 1.1 0.1 -0.1 1.1 50 75 cm /Fm Do Q"], ["VECTOR EVIDENCE"], ["Native evidence remains."], expectation="ambiguous", form=form) + quarter_words, quarter_bounds = face.outline("VECTOR EVIDENCE", 30, 150, 24) + raw_quarter_bounds = [round(320 - quarter_bounds[3], 6), quarter_bounds[0], round(320 - quarter_bounds[1], 6), quarter_bounds[2]] + quarter_page = "q 0 1 -1 0 320 0 cm\n" + native("Native evidence remains.", 30, 220) + "\n" + quarter_words + "\nQ" + add("quarter-turn-outlines", [quarter_page], ["VECTOR EVIDENCE"], ["Native evidence remains."], boxes=[raw_quarter_bounds], rotation=90) + add("clipped-out", [cover + "\nq 20 20 10 10 re W n\n" + words + "\nQ"], native_text=["Native evidence remains."], expectation="no_candidate") + add("thin-stroked-rule", [cover + "\n0.5 w 40 155 m 360 155 l S"], native_text=["Native evidence remains."], expectation="no_candidate") + add("thin-filled-rule", [cover + "\n40 155 320 1 re f"], native_text=["Native evidence remains."], expectation="no_candidate") + add("native-highlight", ["0.85 g 38 154 290 25 re f\n" + native("Native evidence remains.", y=160, size=20)], + native_text=["Native evidence remains."], expectation="no_candidate") + add("filled-box", [cover + "\n0.6 g 80 80 100 60 re f"], native_text=["Native evidence remains."], expectation="no_candidate") + add("border", [cover + "\n1 w 40 60 320 120 re S"], native_text=["Native evidence remains."], expectation="no_candidate") + logo = "140 170 m 153 135 l 190 135 l 160 113 l 173 78 l 140 100 l 107 78 l 120 113 l 90 135 l 127 135 l h f" + add("single-logo", [cover + "\n" + logo], native_text=["Native evidence remains."], expectation="no_candidate") + graphics = "\n".join(f"{40 + column * 30} {50 + row * 20} 8 8 re f" for row in range(6) for column in range(10)) + add("dense-graphics", [cover + "\n" + graphics], native_text=["Native evidence remains."], expectation="ambiguous") + lower, lower_bounds = face.outline("VECTOR EVIDENCE", 40, 65, 28) + repeated = cover + "\n" + words + "\n" + lower + add("repeated-multipage", [repeated, repeated], ["VECTOR EVIDENCE"] * 4, ["Native evidence remains."] * 2, + boxes=[bounds, lower_bounds, bounds, lower_bounds], truth_text=["Native evidence remains.", "VECTOR EVIDENCE", "VECTOR EVIDENCE"] * 2) + rule_word = words + "\n0.4 w 38 172 m 360 172 l S" + add("rule-through-outlines", [cover + "\n" + rule_word], ["VECTOR EVIDENCE"], ["Native evidence remains."], boxes=[bounds]) + stroked, stroked_bounds = face.outline("VECTOR EVIDENCE", 40, 160, 28, stroke=True) + add("stroked-letterforms", [cover + "\n0.7 w\n" + stroked], ["VECTOR EVIDENCE"], ["Native evidence remains."], boxes=[stroked_bounds]) + compound, compound_bounds = face.outline("VECTOR EVIDENCE", 40, 160, 28, compound=True) + add("compound-fill", [cover + "\n" + compound + "\nf"], ["VECTOR EVIDENCE"], ["Native evidence remains."], boxes=[compound_bounds]) + panel = compound + "\n40 60 320 99 re\nf\n" + native("Panel text.", 80, 100, color="1 g") + add("fused-panel", [cover + "\n" + panel], ["VECTOR EVIDENCE"], ["Native evidence remains.", "Panel text."], expectation="ambiguous", boxes=[compound_bounds], + truth_text=["Native evidence remains.", "VECTOR EVIDENCE", "Panel text."]) + add("image-overlap", [cover + "\nq 320 0 0 35 35 155 cm /Im Do Q\n" + words], ["VECTOR EVIDENCE"], ["Native evidence remains."], + boxes=[bounds], image=b"\xff" * (20 * 8 * 3)) + capped = "\n".join(f"{10 + index % 100 * 4} {10 + index // 100 * 4} 2 2 re f" for index in range(700)) + add("shape-cap", [cover + "\n" + capped], native_text=["Native evidence remains."], expectation="limited") + long_path = "\n".join(f"{10 + index % 100 * 3} {50 + index // 100 * 2} l" for index in range(600)) + add("segment-cap", [cover + "\n10 50 m\n" + long_path + "\nS"], native_text=["Native evidence remains."], expectation="limited") + metadata = dict(generator="generate_fixtures.py", font="Bitstream Vera", font_sha256=hashlib.sha256(font_path.read_bytes()).hexdigest(), + font_program_embedded=False, font_license_file="font-license.txt", pdf_page_bbox=[0, 0, 420, 300], fixtures=fixtures) + (output / "manifest.json").write_text(json.dumps(metadata, indent=2) + "\n") + print(json.dumps({"fixtures": len(fixtures), "total_pdf_bytes": sum(item["bytes"] for item in fixtures)})) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--font", type=Path) + parser.add_argument("--output-dir", type=Path, default=Path(__file__).resolve().parent) + args = parser.parse_args() + if args.font is None: + try: + import reportlab + args.font = Path(reportlab.__file__).resolve().parent / "fonts" / "Vera.ttf" + except ImportError as error: + raise SystemExit("Use an installed ReportLab Python or pass --font to an existing Bitstream Vera.ttf") from error + generate(args.font, args.output_dir) + + +if __name__ == "__main__": + main() diff --git a/benchmark/eval/vector_routing/image-overlap.pdf b/benchmark/eval/vector_routing/image-overlap.pdf new file mode 100644 index 0000000000000000000000000000000000000000..f503e5c9ae1d9862deddb80dc071bf227350eec7 GIT binary patch literal 8620 zcmeI2*=`)i6^8HUDf-4ha*<~33t$+CBr7(u*pUPzfRPJ|8p$JYI1V{BoVwgyNi|=2&{GFZi&Q4}Jz5TfR^2_P!&A)!yPgl>c&)2u757X86 z*B|x|ljFdv>8r1H`@45?P=4Nip`W|@6|KOI>&v)m^sxSISUwnDu$kpqg z-=5pU^1Pj{o?Soe`zPP*Z-3dJ4>#A-)z^17r+0@tr25<8?)P^OhZPu-1@*w6_IDr7 zKTe^Ti&}sG?(N6@jehUjBMQoe?>Ol=_|5+C;m7k-MEcv){p+8uZ}|Mh{+Gke{?#9z zA==^m@csV&`RVx6>28nkMf7_(-|w%FyZ`+EpZ{qKNCGUbUQ$6~{c$Qt;@nGc^vt>X z^Zwo8`q}B%=?BC4K%jYJ$dFh2httpZH>8;4UW@ql2RlioNXkuDm!OpozCBJ7;jbq7 zL_T~@`tZ)fl$P_C6qTPZaxICFeGc-)OR$>h!|vG|p0$&8)0_8Xsz_(2U?vyM^yWDI z<}cUh!!P@Z@PD_zBNOiT$Lqu0!~EOn&By8MH@kmM9%bZTALTzA<8*Uug1$ISf8B*L z8*8(zlTBti?t+<}Et!MG%)vTq>U6scezwL(n?8Z-9JBEVXOo|uOEEc4$26O0azSi; zNi1{y%-zw>Hn^BelYqB265Z~8+`Zo=J6Gd;2>Qn3E{D0866*%O)o|vrl1sISmBO4- zFo#;6;b&K*vO?rYkaf$ zS=&_%RIvnV8j-fl#n*zZjvF{){hWQ^qUz_0n&R}F=3G5Cscx>Z7=uhX&movhq*Re0 zy6hWRQnkTOH@j-)5?%E2{W80nqf1l49NuS#GXP@`s=}W_DNb%(!JA!4L|F`#FdJ|Q zb8yvo(IF#MiI}z!d8LLZ5~P{yO-6u%T%nc>82D36zD!`Aq?I_rVWs$x!Uis~_wx%4SoLw$WaSpy(q*l)pKE}F)%)CED#euOTmk@O*h3884hPRXf2FL}&SOZpO z@@`6?qFU;zDJB6>DLUjC@6i zvmYcVa!f_C0xa2zP|f^avax}V#H~f_Pbk<_#s0irv%>-{EWD|bt91_KaIQKk(0HMX zn?|*jEVH2{9Pcd^;JCAvkSY$c*A#ra!MSQBTWh$-$4X*KihmYwq()ttV}cG_ED?hm z(BD*oJpm7Sd)2Tm%+XW-ggjs$4?j9s(K<58<$9r?j$22_`{*1RHvS=6kjWCrLSdbN zTuo9OZ_t|yVGSustqeKBk<-LN30~?aB$QLAl=Ka%f=%PlP_mH{$LVo1M1;{(!J$eH zVWOT<$f2Ybf~eiqN<-n7miIIf!yW^bI}*kzk`{L0(BI4_spXxzxeDP!|1HIa9G+lF z)imxHQhU<$Qt5_<3f0MHH9oATZdR6dLk^O1_(e&ryr?uBk)Z7{P)P&*$(o50IC=Ip zTJgsUTO`3l6#|kBE0k57h$>!4Q^JFoFodB*ji;PhV0Q?b z)ls8P3!f#3nB-PB3(*^65<#fM5SlEK*&>IMSvVCZi%3en!OGee1C={0{ZwlZ6SkqWC5Lv45_-5$>8G4MQfLJ`aoT{N7cbHSE{!okLqi*5br>FQ#yAi#I)*x14}(Y^ z(HLVIY2H9n#T!zq=&Xtqe6drcEnljDFa-zXG-$EBDS1mD*ZXfvLw%{O!XLei$Q^+O zNSLYfF2`JNUl#y5@k-+^zUO?Bk_#V;UOvzQv{#k_E~q2&q}@ZJXbm+M`4fJ|yCM-^ zyR;jE8|y2@RP+iodGsbW#2Fpvv>HkiRi)1h&|6%JI3jfv`2$?C^pf1M;3A*GrA@Y8 zESF_+Su|4sPFtrMc_}*lgAQ9f(ch9R)UJUyKlVI1KvXiI7FHk+UoAH zQu|mVk(S{NuEXd{Qz7kaG`4B1(KWv04t1LOibaW3Nl=Uk;qaovrDP6|6ALlJ*@%>- z6P6J`QRV6TgjOkGw)PH7+`#NWDuF{Kp4!G?$+13B!JsJ5 z@hz(Ds!vDqCO3)n#e^|v^RE(vkZ*=jh4Ye`Hif$u%n04u0Na(bT z#UlnWIs9~)s0NP770z9w-@?B%$CPZ$^%Sn=Te(O2Q?czj3$GxTts}3H+2!UZaW&n_ zC3$J?sR8RSyun-kD%CqxDM0HigD=A&O;2X?(2^YkF&U62ZMfElEi+qMCdm!XDAT8^ z%?`s>XqG}IGqOJJ40$HI4I)(ImfkxHUw1ih$X2lKRa#L}kO7K>9p66g9Nn@QePFjM z)=Z$k_z2M75}OdstBfPActzoOpb@JqMjTafX<9chtz}{aY^Gh0J6cO>vrHCJ%IWwb zy`ZDDG~TpL0fpQ$`M^&%JLe+VNKYhYVXBTbv4j$oDaE!02=jL*M~%V`%Q~swaXC#? zRw+`Y-Ep~DmH?~(XxFtI(iZwiN8jWaQbWhlw26xPw`w>HrOYuv#8B27xcJpsmh{rm zZ0d;E8{I6Te-bMYAe)Ss;wB12beBsjjFQZyF$z{7AD7Jzb>uQ=I(ju4392lqD*bPp ziXGi=y=1$`Pi-T&K{tjSr=@>jf`M)O&865r3JsLJf@HR4T$WY^&_w@Zw;)?e~&akw$SajvX4K930uB`^>)>6Q*$igu60k!r< zVSI+Q=>9U%B6VQkT+XU!S=0#sEhGCD-=0V6Kwy#3PvKoiblMOi`@%Wpwtk37A;a$ACZa!%M2d-A1PPTM)*+^b?pL`z+i$H|Ny8NVPf|Qrh zABMbaKOIX!XP^V>Jx?y{PbMg0jRcZ@g>IzpKWUNI_G+MHfcyXf80mz@X)-xqC}3TI zI_8&UccFKn4%Q>>TN5e-*teiCFji(L6yer2=mu4@q{ivjmqn<)SxCFDLbVw1!nzA( zUAPL3&zEJWR#0{deH*IHF;L8*j5ly?Xl^ExnKranc*Ttx4Vqk0OYnh>!grf~$vXY5x z%++Ecmh_*X5->7BvzCMH)M-LPCUEKAd2iW`idWf3w%?0jipz4;(MC%>kO^>Ijyfvs zn*Zx|RHGbN_vRa%NBX#{E+!WGepJhV=%a2aiK%kShE!`O252i%ZHg6?Six$Mm38mT zE@$O8RAQh;OUs(Hd7~uOY7JfYq|JAU-UfFd#Ih)DZZ0NnSEV{A$kffQ)X_V(t;VJV z9d~U0WdqX&TO?lxPK|TdC{3*6-BLuK#8qkXw2`gn75SgQHzi2j|0J$TkjP0N(gtG! zXe~_}yp<+pfKJs4(3%TXKl;_|ulM`+J2T}S%c|v{DMdQA>HRdYI{qnW_9gBza_5HcY&dN0H MzWCzn?_Tcy3;7UTQ~&?~ literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/manifest.json b/benchmark/eval/vector_routing/manifest.json new file mode 100644 index 0000000..150df9a --- /dev/null +++ b/benchmark/eval/vector_routing/manifest.json @@ -0,0 +1,468 @@ +{ + "generator": "generate_fixtures.py", + "font": "Bitstream Vera", + "font_sha256": "c4c45690b345435b2cba52ecabe275f05e49b389b39fe68ad03afbb551288d3d", + "font_program_embedded": false, + "font_license_file": "font-license.txt", + "pdf_page_bbox": [ + 0, + 0, + 420, + 300 + ], + "fixtures": [ + { + "id": "outlined-text", + "file": "outlined-text.pdf", + "bytes": 7865, + "sha256": "a120d0a06e6b06f9046cf4411fd719337cb868f02a7154c8751d1c684f2d0568", + "native_text": [], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "mixed-native-outlines", + "file": "mixed-native-outlines.pdf", + "bytes": 7933, + "sha256": "e20a70fdd29ca6f743f26282b5f719e1af3089f64902c86f3c63eea62f578b0c", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "partial-native-clip", + "file": "partial-native-clip.pdf", + "bytes": 7972, + "sha256": "38ec8304c2a3ac6d5c405cf890074f27ff5e4deaceb21c81a6ffbc4844d025d7", + "native_text": [ + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "chart-labels", + "file": "chart-labels.pdf", + "bytes": 7878, + "sha256": "603574c63694ab829361aa2628ac9f5ad8fe433998b79451c4f4269e29536519", + "native_text": [ + "Chart caption." + ], + "outlined_text": [ + "VECTOR", + "EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 160.1875, + 139.660156, + 257.875, + 157.8125 + ], + [ + 162.355469, + 79.660156, + 280.644531, + 97.8125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "transformed-form", + "file": "transformed-form.pdf", + "bytes": 7722, + "sha256": "bd625fecad029ee802aa8065f201dfdee7d40ceb119dd71fb0e06fb7e26955e7", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "sheared-form", + "file": "sheared-form.pdf", + "bytes": 7728, + "sha256": "ea380cd9fb9a97d639fb12d5a0f5995c8afea9ed8a211c227ba0c1ecc48f117d", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "ambiguous", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "quarter-turn-outlines", + "file": "quarter-turn-outlines.pdf", + "bytes": 7899, + "sha256": "ca18e1000763b181d27ce66a388611a58327ea00094902284675974ec6d09cbe", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 152.1875, + 30.1875, + 170.339844, + 256.839844 + ] + ], + "pages": 1, + "rotation": 90 + }, + { + "id": "clipped-out", + "file": "clipped-out.pdf", + "bytes": 7956, + "sha256": "03447a3542f221f3d564e59b0865205c21416a6f3031c963de9a8909874c6347", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "thin-stroked-rule", + "file": "thin-stroked-rule.pdf", + "bytes": 682, + "sha256": "cfe11b4ed83e1d47ebcfb7069f963c2536a757d356878da67c949454ae41c3d0", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "thin-filled-rule", + "file": "thin-filled-rule.pdf", + "bytes": 673, + "sha256": "221bd1b85944d7ae83ab85b3aa195595389ce956ac43db6b35012b481a1c7f92", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "native-highlight", + "file": "native-highlight.pdf", + "bytes": 681, + "sha256": "75619070767e8c20428e16b439b9c9298f890626d07643400ef2c4ca72afd32f", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "filled-box", + "file": "filled-box.pdf", + "bytes": 679, + "sha256": "c358d9bc1bde5da74675c410a49f5a01d1f0520b7e00716d295e2c04bc3b7ce3", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "border", + "file": "border.pdf", + "bytes": 678, + "sha256": "28d56dd3ccff016388562404f0f7aded0c68641ef2ed8d977b7af347b4f632f7", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "single-logo", + "file": "single-logo.pdf", + "bytes": 757, + "sha256": "b2b4923aecc5e9c4cc4fc66296441a2aa72b389301a1ab3c48abb272d7ba2418", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "no_candidate", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "dense-graphics", + "file": "dense-graphics.pdf", + "bytes": 1636, + "sha256": "f070262f60eff6933a3bc616a6ec0cad71c1cf011363dc5d9a713e785b4fee0e", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "ambiguous", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "repeated-multipage", + "file": "repeated-multipage.pdf", + "bytes": 29417, + "sha256": "9542b81898fb78ea4800f1301f843cad1617ee1cafe69757f0ff014f72204cf4", + "native_text": [ + "Native evidence remains.", + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE", + "VECTOR EVIDENCE", + "VECTOR EVIDENCE", + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ], + [ + 40.21875, + 64.603516, + 304.646484, + 85.78125 + ], + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ], + [ + 40.21875, + 64.603516, + 304.646484, + 85.78125 + ] + ], + "pages": 2, + "rotation": 0 + }, + { + "id": "rule-through-outlines", + "file": "rule-through-outlines.pdf", + "bytes": 7960, + "sha256": "11a730c563868cef7436e454523b2020ad0f2ce409cdb69c6c5c90edcd77b5f7", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "stroked-letterforms", + "file": "stroked-letterforms.pdf", + "bytes": 7939, + "sha256": "b498d1e3a1e8a8f07c17939614de0e9dbf4084e53e13a0e475e42493c95a8253", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "compound-fill", + "file": "compound-fill.pdf", + "bytes": 7907, + "sha256": "c50de265482f539d7c71e5e4ad0406e7cc72c28909f23b3f536fd448a1ab9d7c", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "fused-panel", + "file": "fused-panel.pdf", + "bytes": 7978, + "sha256": "5467f456c8a6d10f8f0f8ccddfe46a1f582a0e04378f7f46184b87e4346b8c37", + "native_text": [ + "Native evidence remains.", + "Panel text." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "ambiguous", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "image-overlap", + "file": "image-overlap.pdf", + "bytes": 8620, + "sha256": "afcce11a41cbcecd2d736d7b2aa4ebe1084e69f086a6d4441ac2ddf60b0e14de", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [ + "VECTOR EVIDENCE" + ], + "routing_expectation": "candidate", + "outlined_bboxes": [ + [ + 40.21875, + 159.603516, + 304.646484, + 180.78125 + ] + ], + "pages": 1, + "rotation": 0 + }, + { + "id": "shape-cap", + "file": "shape-cap.pdf", + "bytes": 11699, + "sha256": "a9727e772c7976d248dc30bbde3daa3130a9970cf7a91f920a52c8250f212c98", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "limited", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + }, + { + "id": "segment-cap", + "file": "segment-cap.pdf", + "bytes": 5888, + "sha256": "c58293a2fdbde0c2a66051ad98ed448cdfcf5829693797bf2f57f3555aa639fb", + "native_text": [ + "Native evidence remains." + ], + "outlined_text": [], + "routing_expectation": "limited", + "outlined_bboxes": [], + "pages": 1, + "rotation": 0 + } + ] +} diff --git a/benchmark/eval/vector_routing/mixed-native-outlines.pdf b/benchmark/eval/vector_routing/mixed-native-outlines.pdf new file mode 100644 index 0000000000000000000000000000000000000000..201443e09b6065bd00e73156dd75123c90b079af GIT binary patch literal 7933 zcmZ{p$!;CT6^8fsDSBfdS)^I>02l@$M?!!MHf#b3U}SMcUC9(U1mu;L`tG+ljvJRk2KKTKCY9zUF( zC&z)GrtiKxoF3lELHT+AK|go(x7&Bm(=S1e|A$*1KRui$t3LEEK78}Yk*l9Sy*;;w z<#{_@y*fUh`fvVvy8C=O-`*UjtJe=VkMC|Dkm_Hz4}X4mzTJQ!nLPbcGSTDd#6w+? ztACu{-5y^({xd`|07s`PV7?;N-)@^ZDs?yg!)f!{OC6kJ-t(>H0n95}E82 z%;ch(uJ5Nm{PTFe{d}5EpKsrt9&S$4)9Le=x<3Rn zJ6m#`Y%z1N&YC*i9fF^&@zJI);5x@_JdfMtXXjE(j?*#CCYoFjOJ5SpT)%R6w6hH^ z=F%kKt(`=7hhGox56RBeI3I$(@&1s*Tug~|2j6Kpb6LryTEt3WPAM2fw3lEvn-HoK zCAiy{kP|BBufwU%*%|cT-+c*ovv}D5N4PUx1BVI<&?&yPvgP#XuEHpr#RN z%Upad*y?@d$uUk~IEP>_JudS185Fts8iAD2XVGp%P{TE@2L?8ZSCzq$&~9 z79y|I5JiGCbG^w3P>?Isk^uw%ipiG=%#(@|M>wn$A5z$XhMQsx%lGTqgt9f|5asN0 zX^M03)grZep71f&C1mFPAu0}xCAoyCLn%C0vNyb?3@|`07{(f~GAp+dx7bp7Kwh$B zmd=d_Ura_hTk^(-IigWjv6~T=^dv&r_w;6eL)o z1n0$@h*eCDH~>ukfvZ|vWG9g|*@%KEOID+5>?tT<>NJ8`*6bP}W{E^oOF(@V0h09+ zLp(;*5IN#3)509kNw@|PAlKO&k{sqiTkmZqBZ#nIb4@~V39WpvR)bjuhB}-598$_P zPwG};&_tfl?-Q&zz!SG9Fl6Pc;$wmwl~&{}yod{x0XD3KMKbb<5NAI~P~@13WCd8V z6``8>y<}qp9f@0u*q>0asfzu1y=I36T3C2fC0FYl$l+XdRG{%f7dMS+D_LekOE}(J zD!~20T0*Kg$X-+M-45rfjcl#q9v>@-DJlL{ypbApWsV6tY_UWPYCwOf1iJtad3)8c zF3iz;gl;mxS3>^i;>aYI>qRa!a4!Dnxr`1pf_{|Ye-3I zWyleZoF*1Z@KQe^p`1d^9MGr=HjP6=$wo>Xr^n3@5k^l1hblRQiF!sMhmu+dqIOp+ z4TWD?-qR!udkj?WNZ6@JTG)X@e>1;HE$`ILRR|yYZz(q9@B~Y$rg6uR+LNYhr5hS5 zR41R+_^_V3Sy|c*IY`Rk7bUgwqSBDySt3E(W1x}-`Zrr9O5o(#*J#Bb8*GsT4^;?A zGOSQmaU!aCAx#MnV!{xH5;dN3W{sg-TABbtB}3eT)=o86Sf_VgJ5)!FIxT#bAYzhR z-7G|Jj7bEc5<_URNM?&1N@n3yoGc#3WSrIj4oElTL& zLZzQ__DG=(?8IpUeqOvt54beO1Pu*sjMZUyxEbR>z~~t2Y&{Gjc|>E3X{31rO%-oQ zt)jCkQt-u2k+yuP0>Ts=kkg>W@}}f1eO&LqEe-WrTZKP*8Id~z4UjNX=UtAu-o7pX za^jW7U3}+!l9CG_i(Wp^0<>3_0xqZ{@}%8EqG$~@7Wor?#=9aBpRMhN;Kuq&F%`W6 zO&-084RJ;XI<1D%L{;hY0`wM_B92HMMg9PnEWIRmEV#(0aA}jR7t3XtTo%m~fYY|A zMqY{z|DeMjPxQAW3$<(D&5ym)Y1P$*KqVEA<$(9l0S=Mog|@nTtkgc%NTg+WgX=K* z(o{%08;xxmYjle*xkH_1z6wP;NXCqRUPFO|&kpuatL9P)I zx|inF7Ja?ZAS5Ndq)vY|fEyhu6QkrZu1RPQ)02{|OO*LS8+MOJ85}S z0fy*~NHUTjM3txS6I!K&+1fiSaRajhsRRy{cxoGmCCBkVt3Dmco7^PQ z7Zb*y&A&k_W!Te(O2Q?czf3vVEoZ6mLc*?RLuTurxfNnVtt zO7%`v3eY;s;LC7G)05dev}DIXOa|mh8?NtzqqU?q%VZ&?oQ^Nj3p!d$<4xNXP{=Kl z5BzjmUU9UV?9k&Rw+`Y-Lc**O8`~? zwCh?9X$yU%qi=EysiEU&+C)YD+cX@8Qsx*SVkqkkT>R=Rn^5U!c6CJTjcyjvzlaqG zkWEHRaTf(5y33^%MoH$<7zHbkkIQC20LBVHc_F zz>i|eVXz*}3V0FCbhhhIKYoXGDH~2qv$G=6tZ@9aF(D1jEN&M`FXD>yXxJFR+sL>h zLyc?R)AwJ*74M0hVQFo#=*EXTT= zIjf>&Q6v1fjO=@Se;%m=fki@J!n=^@v?E0Jg>%Yn11TdBlOkpn^%UN}R4Tjx^qYqD$>_l~Jpvkrr-Qj90+HOhC6CF_8 zvKE#1oTtN=8b)K?_o5;l)3CIHfZJkJG%>KX^tH^;OTy6xFj>h&Hs)%v5KHxAo{3VN@A+qvLV&li2>S*RGVT0C04LnWM$nuv-Pa}g-Q(6XlYrK zHgA-~TCJhmp0xQc(c9n-gjg1(&CSKc{i;+41(~|rl{$LIw$<2`p!)-xf7!sa!4}E4 zfm7q$HA)lfc()YMi?}LHo;I@eydwVvd{=_h{TFdnf<#XGkTw_-KwD|r;jJ_&19YlZ zfYw~7nm+(P_di@Ne+vHX>GYmIVx|Mjs^z~aMLM?W{WP#Twckw-10(0Y#OO_4U~~sB zFwfr(wr|f*$J@KpQ~Trf&$s_SO^LtbK0cn?A5z;Nt)I`wr*o4t=THydeDnH;H;3N= Drz}JZ literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/native-highlight.pdf b/benchmark/eval/vector_routing/native-highlight.pdf new file mode 100644 index 0000000000000000000000000000000000000000..58e4c662aa3d4292815d7905f61ef5b44ffa0055 GIT binary patch literal 681 zcmZWn%Wm5+5WMp%=3*c{)D-n_Z6F9xCpHT7ksuM!0znS6v{q=?WFgT;+ON;9?AU3| z1EC$x?#`@saCbBRXojDZL4vSYsmn{~^3QLAzP8b>Lj!$h8|siU;T~64ihM0ZiT_R5 z^Ym9&cX-T&{=`+n)<>9kiP>4=JTOH+Y!|U_EbR^S)ONIg`9kX*MORtqnXf|aJgfTY z{Ab^}BXCkCtS9ZFqmtpcK!a=yQ{+e|3DN>B zVW|={YFe@g1D01Vd5ogfAm?ku$OacbY;-#U?OaX1B2L^`=eywtlq<|iQE}*V_-`*Q zN#&7Bjt8Tn$O!NLAuQQ*F)nG8YovT-A_p{gl literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/outlined-text.pdf b/benchmark/eval/vector_routing/outlined-text.pdf new file mode 100644 index 0000000000000000000000000000000000000000..7159244cd3649884a645950618e853ebc1df090f GIT binary patch literal 7865 zcmZ{pTW=-B6@}l=ujm&Z)2{nPLP&vtA|(+BA|(>agYg(J@{FUgiAa8ZzE$0)YxfME zhmp@Z)z$mD_T_x?&7Z#dgJ0{#FK&7I;g`#&pDs6l|L?!=mzytcUv3|tep+t6x&7(> zc?lf&cKPhH%l*4II;hX*2exxJe|dQGynLT@{9oMi^y|Bq#hVA?M-P7W#F3lte*NKP zyjZXI%gyJv&-dFmf4+bG&;858-R*Mo<-5D5HxKWS>K_m9{`l_s;Q*#&diBSWNsssU zyl7M8=5P0J9&SH>`fd5%aXtmNMCTCr+xzFIU*F%Mu}m=p7=QRhg3AKS%|E~X;g|cn z7oE^&L0_N)w?8j=KXtW}K3}Ez>;1c*UVdI`s7ViAo?qVI-@d)1wmR>t?~5;P;j>$V zZ#6Hzxi$IVU0)tADX!kd?8^smUvqWQ2Of@V2ql-m>0DNqU2Xn+I=8j|&fVFsK80M{ zkbrlcBznC3eEIQG{MucJDcKv}UTRvKYq_7`M+;XjYqfQcSS78cB!`IpH8`v;r5>aN zcgL1$LFM}UaOrCe4*kz}zXpdjh7>R~cV`TgN|Va3z)qJ=t?|w2vvrOP)UgC=I!W8s z7JI{1Z%=T>`nATyMLn(^HRbeL*4iUB>0#};IfqQOt|_^i^U@_j4mA$2mF|;Y?k?S} zEr%TSd|N}WIh3Vgju>me8Gv~Us>WZbwV+!M@b1zIQI=CD%qHB@nnHI`I#i@;8Phf* zul1BAL0P%pRRn0rm3pgyiNA7*Z2^mUu4wH8xJC(vlp`t=?`2r;-Usc*5We?ma42q)x4#kz_QT@X1Qk91Tjk{nnnWJvj|XpR1EPl zqK3$kv)l-Cz@TtV5}@`qI+7gbLE9L8B_oKihlGaZRPvUKevK zG3YX{uNG+ z{88CBz(C^mCi@c#zI552xAz*bzzB=zx^lIzi5#xoMgL6S1xF_aLVk*Ud7aycXU0HL14o55zgBGw)8o^$Hr@UL$@FA@^ zMuZ+Rz*i!FbO~fqsQpSm8}0)kA9Dz3IQWNXK_*Wii^4Vn3caW}-k~>i25YD!jWQGn zM@|z9Ek)H&NT`<7DYqxo1e?X7q2eVnkd)IeMrz|lqanewM}oD-M3n~inylJ(A#|4go2{8f6tH zqDmBLDm;h@M;Iz$*;;jT&=Wd{!W0lH1%YqIb>} zg3yQ|nrzB!$zfy`r{ZLZWYn9iY;7@7-Qnq{MuTX`T5%d7evdWA&0A)Z7vw?`VM&gf z3VkJac#IH{5>Z9it(B@+r$5?C(qL?{oAP3!r>L>`w4S-ySlY;8-C~5EE;RZXXHQBw zz>PS4!q2-G=>hk~n4qDd&AA5*4>w~R2pAnxUwwo@lt(nim`2STXz6%EwTjNFso={_ znYMhd0%8gd=rm~ayi<8gAGh`2o`(8bTg4w+8POey21v}b@h->0*1qlla^jW2T|5_J zQOSjmrB_U}0OKut0T&vGJhgjBl)at=>G=`Zm@m(ki^cbr^kbD%8$q=ZD4? zJ>pAunA6Nxsi~tBiV-0k-WYHxnZwJ8geOG^9qheq$t1;JrGGw2|`pw z`aaPrCCvA&!;%}A9Y`f`s3cO`IIJAoCMp;dOt`pV7F}cCHYxZ0Cm*$v~ zjk#XK&3vnS)St?>$1Hq+Ty~7SBD3qwui|F9)g^Ur?x_LCFnq#C{up&M31@Ri_<>@6arTOf#}Q9vpe5-3AeAaZm4E z#n+o0IOHtY=2cozQjh_Pgq=^ny#%^t8GT}p2R2M#AN&ZgPst`k^C9DmE8b#oJTQn= zi;+N8+?&=DOlw&f0f%Xid1A_4D@O=5}1!vRsTDt;z0M?uh|~))7Z$J(34^3Y4s0GFtF`* z^Jvi2lhOXg)IpJBA5d`n(ZLO=b7l#r{GTT5AxW(Fmk22*n*cQzO_$Zp` zY_CK8{0{3ePzWIU0f#VzmI`(MQk?@7+I zx3*aH;KLIxJ|x!;18mkZz_iHDF!TYvZ;itI4r|fR0GS zw*4n9^3h&Rlnl@Z2*9Zmny1O+dZmD41sa%Nt`jQ?X6Ss!u zW-^&+!+J%pyKY3S55c#UsP*>TiQ3pelVd4*!p&B6+)_d$I-t0HEvom0NQbW)Mq|D0 zMI{~6uv$UDV=*dC9Bi$=mKl0aINAUvD@|l`u9k&Z(tm&|U^GFqmV@ogX+lF2xVm@V zx9>*fRrZnNcM(i+U5*CYXw?HvfX8w)P-&0+KenS5<-oBwKjFM&8+Xmc#KN{8wK5?3 ztXoE6s@%RIwc5!5V?}CBae#^ytd?0>Z=KootbU=Afd(z@YtrG3R#>YI^w^UQ-xYcr z+<_4LqI9^qxqM!g+Mpm)PrK4U?>M#^rxNt`!scHam@(KQ`7v->oV#XeVjUm$BKj(B zN>ikb96hh(e+564Aann#xG6!BlRjh&#stt&nojsAO~wElRU5!+E~@6o+vn{^$NgvA z-`?NjEk7<3>&y6Vc{ed~?n8{;7zkKq^mtTK%`9EYCG8q5> literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/partial-native-clip.pdf b/benchmark/eval/vector_routing/partial-native-clip.pdf new file mode 100644 index 0000000000000000000000000000000000000000..7d672e2b60bb08bf3330e2d63fc387f3f1f893a2 GIT binary patch literal 7972 zcma)>+ln2>6^8HoDf&j>T%@PY2QUW9kq9KgjxBJ&H1TJ2@zyj}P>77k{~a_c;F&Tbu)2Fvj z?ZxtXJ72uKd_47U{(QRm&*|y<>TgSS)9uFs8 z)D^k-+v(l)<;(lu=3flwgEO-?27!M(J>GwMxI$wwMH8U?VP_FsEHGdEDTG~lJoVVoIzB_(sE-%StZQB324ZO2HtaJqNqRgixI*!QH-u zoKQJ`8BTS{&Y=JC?sKqPybBIPb9cl*Ar(>i3~c1mp*6laeAdnp163@6nnt88OYyZ} ztJ?z{v3|}za8dP3MNM&fPD`#Hn^d>dSd2lYoR<(xCjP2O5MA~SEUDUH=c}ojr9>CK zJYN=9b98Agn8W+*a0X!PK~?xGl;Y&p9lV-KBFbW@gxP>gSc0p@iw+s7O2o8<$SXBO zksvKxZ!!WDSCvvzkYI%poEL8* zRxvf=05JIjSGBmvP9h8PLcx?Jt5G%f6cjLZ8o?}Eb`21-M53uBpgxNL$$E()UPjaq zIpQqS!W_^^xCRj**Tox>9Ogk=?`Y>Q-XV zL|&oaPq5+uPu!xwkd>#3j|p;AT9LQ#A}&-0*s&HC$;cx@oc$m{kz*>76=2C$glgvZ zl8p^?ByKHYe?q~gD)#5?H9IWO!or&>xmuS%4(F<)0*zO?xN1~e$ub*S!tuVf0^Cm4 z5>mxM_L_ok4melsWNQug_*h9yN%1e@jnt?sOH9yVizQ-E1NuoN*co`p+pC6kVTs-& zbdv$T67okEM<%&kH~Q(gb%eZ+&Y@xBAEE`BEP*T()(ObfB*pOxy`eK$LrPLBLymCd zG_g>Em--0_fJRlYX&f3#Hd5j^J#L1GFnTIDRLLPs)H4b>l+;2HwYyqrDE!j$ ze%AIFsN9jTQ<1c=1Bd<=zDX_b)Xh~0ANp@8HstUGORA=E$B^2Srdy>O8Y)yLpVj!V zp1N6C+6_5K%HbC!weq6Ukl;*TA+NP>qd1SA<&D62RT zRlJa4PeG+89GMGhsia4Jp~k(7Fam9;GfDtB1=sn#GWvX(gY5WmG5?dG1@WB|Dkgs?=8 z91?vccUX+zkrGjbuuDl6h4SrsYx zVy8%3zE%NY3J%C=&|-O~ z=X{cq3m=PKKF|WRx2y$RP)FoRyN5*48fq-^C;W_eMIs*C+6}>t^_5~OdIg$1dJ`Ms zf(~?A4W)^y(&q)}Ev`izkvfX}0WMj4N$yy2kx${$CR;C-%QCsFnkfLM?Ng1sDLVXv z4o5uE-;ylUu7Ni{_D-i&R~G`6R6Ldg-a`jCM4A`c>h7^p`&c89mf;Pq!{}>MA?<85 zwrQ-Wdsm8kdGSV8WEv;X* ztHdDWn_-(>mC#vQD9QqFR0{5fI+$Zej}bl0y7n$4bXv#a5rdc==m2fIMl#wLWZ_+0r^m9&kpPK2;rd7`8&Q6f&8S^>Js&Gudqr zp&GaJ-dXs%%Yj3Vf_1Obijsm1P$cYl`gU@3%VP9_-RxL1fqw8Apq~<(5Y4-cBd&Oh z!tp>OR#}WVs^Z$T9$;F_#0c0-yB~M7megjMETojv@kM$;N2_hTX`2EHxn=T!pROk7 zBH2h!BxYf%jy3tnY0AFrg2))kMCjzGQP^Q!C-pnF(?n&JB30TQ+s(2BUno*QAb-C7k3R-EUwk(-f56Mw%NlYdi2!OgRkJ zqgesZqM6Qi8|ue*SeLTl#54yh63q(7PdgLR&@AG1k@PICNRNh%5xk9z2Qt*S<~@D? zSzPg+$Qjnw7K`qDc)*1Z$+g`8-C7D57FijFKA_g#D2(s07TsSaTBHsPoXc4it&1Ar zzjb6k;>Yty9SAHE`W)VcM5hBGvM*dxZW~A$iI@~ItEi{&{-jdj8KB?UxUNu?SKh^| z`{t7daNuh7>10b+*Nx=1`{a2nECLZ4`|^{X2~u81e;9Jxema(d&OisS7q48`pG;81 z8VMx*3f)NGf6^lF?bSfZ0Qmp`80mz@X)-x)6tJ&A9rMdJv2tMBfjU@^v~NwQ5MbYe z!oXOWp-_a|*PsVf&5{}yBJPV&eY22uUxjKh;Dvn`%DQkD8lSJrP_3Zs6#6z)n`5Au zLm6-2+R)rgCNphlugL4Rji~k^cwdQXZ;zd*jtw-~m!bz;ZAIHH$$6p!id)yB@;>M3 z@TG>)SogiCNXIlRtsvmO7!^$nY%P5)GxVBpv;j<3GLenBS}eqp{xehpMkZ+1ag~9(JXU-mz^p4khSzV)HK>m^RoV`95%JoV!M8 zVjb_+B6=2ArODGqww_nyKZ74ikh=dYu1b)|NgvV%V*+R|O$WS{CS`z5)e6v>3sv(w z@nipebN$8mw};bve%4GA%c}L?oFX0D{C*x-o!fWw-N4AX&oO$FXBgeVGtBeb!T#*i v!{znO>7o6`{nPdTPIKbt-uwHf_S0(nZTs=*^5Ln;2``uF#fw+pzn=aNuxLsc literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/quarter-turn-outlines.pdf b/benchmark/eval/vector_routing/quarter-turn-outlines.pdf new file mode 100644 index 0000000000000000000000000000000000000000..3e18a48fb1795f5d4fbbbfa836e0345f4b145d5a GIT binary patch literal 7899 zcmZ`;$&MVy5xv(}^rx`gK-T#EU)p)sKIC`;D9P?!KAs+p z$AO>Bci&B?yZ1cE&-(|;xvM{)-#?mP79Rf>Zn^(_cQH;q7@j=%_8v#Be*XOKA}{9a zow<5_cs!MF{&c$ea=M(42Xpo2?s)(HdeqVsU=))&y%I@K5)b!{uZ)i+@j#_n#k*P@b%aEb@;t)PZD}tAGCR?$hab z;R${w^gYzUiylp^ry6Da+ywi}>F(p@SCjpg`QY?;c{m+zr~lxHGvCk}0U}v*yq&E1 zIK94x0M0mXu0P-azkw9icpSREHNX4Y;d1_RGN&)+_out#$vm8H59hnb`S<4flX-JJ zg*2}zyLB}#T5~%EJG(VU<5HfNE!k}`H&gI)O4h~n6}-;g2Ip})`Z=eV@bAsEggM8& zLZsa#OPc*NS8sES7N_O?c-2mxo9Wl-!?dh(&S43VaY%JL#cf_g-1hLz4vurQVY3NR zMK`;}QbLzrfbAUo;@!s6+q`5Cjd1#TIIeSmC4$@E{SxfvZG-wNPUAwmBq13>JqueL zQ(D6^yU&A#LQu}Ls7b~X)LhBc?WEx_r|82%Yi1Y%YSQFsnZve3D3K%n^Dvu$+X7)X zlNb1a#bW^RJ&uFQaI@?tU z`sCh=0)h@1w&f+zK3HXUX@<3Y=TsHnc`r8AaU?DV8tvzPz;CFR=}$e zh45iKKMV$xwuQ}=X(24b3TKrY02GQnFBndPcZG%VTfGB%GpeG62$5mWmxZvca0GxM ziAm`lVle@$?Zi8l6%GbxTo9pFf*ab_NF-R337Ba?Hb9}4&%^)JS@?Aht9TM5Sh9un zXp-35K-vU~ypsmuY$dET$O@G)O!nHlJ_j5OG!lf*CLg6XF+)N&0(6^d>M- z9V2GpAFy*^0^yCAXF4kxf|0X#jFC2ul5#Urd#Z8Q3&HKQ(W&^byt3B^4XfO`B9|jS zOEq~Omc+Kgk5-9GmY|zSeFmpJfoqXofV-q6EjsJxrKJhvBt}z2UUU|YU7ZPUsQ9+c zf%zdrH|8I+A;U}d0uW}n4iq^~8>kk=fg*$nZqRcnYT9{Q#PbHG(~d$DVirYhG3r8O zof>fJ-EiT`0bV|2WY!Ur8m$f@Io2eqlpp93$}h8%`i$sMr$K4ROn3~Xelw(8oOLCV zx){!8;HVMsgCVoC$ce46b@g_riZP9&B39b;&5d4XhZM>XNcvsmVlb`Ru(gvzz4DYla z*%YB7_e$`>w_{9_jzkDy3l~I`^;$Y0adH`neM8?-3XY3Vj+qOApg`p)3LdnnTL;5R z0+>HEb0}$t?GQ;#ua20m$F1FShIlC832|FfnUIE`_l9lYqL$xt&0E8@A z7!NJw)*Czpok}x78?^r3qa&4u9M_>SlB@=7ancMC>kWkS7JZ(S7pl}o0idk7QhiyL zq`gELnpuhVC}el3w}QB%o`!T9XHjr&LDwqtNL5P#uvV`{d{bP}sxbI84lSPIV!_sR zqEeTRNsJ|ilW?!yGMR=mS9`<5Xjsq=usH{MRJRm`HQLKixuTy&+ZClprIh z$E-SyPQA7i`ex834DID0$&yFO?2sCqc-8?*Jv1ClX+8#uI~0EduN_HP8_~)+!8ucI z81WTuAiq(FOgu)EyF9Pdjw!sSFA1{5be?Mq2Znm(u010c9V*vev8^Vro+i z@-g+wM73gFQkOfd8YM4k5cRyQhj_&}jHpJNAd;$!(fF+%jRF`|VzBjt10x=J#}FEX z*g6|wa%c(;ng$sI!wrhMfs<~B!I;I46Ywwu$Ofqva~0>(fs<@hg3`e#oRV{XlBmNp zSThN9ZzUdY?eik|N71CaK*=)O66zt2T|l8vqwvN-sydBo&Tc@ilqQT*a76xPM@s?9 z+(BzgRp-zs;@13%mQF-Kr$W`rDu_i_Gk}jIaZpvAcmSFs2^92fZ7d((-A*38HH(~V zj#T*VaGm4W&(R@E*FAEd##Nr?e?x7|!0m8b}>c?bsR15>_oXy+`|4`WlSVWv}Tu{S{$kmM)9U9jbjrorY)C zNo3s1VY+NKwG6scX^9cUb+@Ut11b)!h-gr8lAewGK~S4X4&1%3H^omZYn7mVzbR(( zFd+h)-c>$-z`q`CdN9w}Vc9evM>xUNUIM;0#N4N=#7a?0Ov)+0?o$_?{U26JJJtXI literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/repeated-multipage.pdf b/benchmark/eval/vector_routing/repeated-multipage.pdf new file mode 100644 index 0000000000000000000000000000000000000000..167e7fbb4e9b213b5775bb8ac9704be25f73a3a5 GIT binary patch literal 29417 zcmeI5OOGYTafSEySJZ}pz3`oSKLA01DU&eZhXiQ=8xXu`_B5#`dU}lMCL#OR>pPiQ z8Id>Zt_*lpTo7~5RAog*#w+7g)_n7afBOC(y6bZ3nzs4!!_SxB{HFQv@qhmNsrm4` zhxdnm+?!&)4e|l^FJci3>{r|rh>&q{% z?+jtL57q78f8ml3fBNNz_waZL(KR2weRz9{fAh~zFTXy$KR-S+AHI8i{POAf71{pd z`Sl-P-<~sI50C)0e45SIryz@_^Woo~K0QBt`{l3ApIfdU`?eX{7J>ix^!DYKuaC@I z3JC&)KXi=*mmZoA|M~kLetvp<*A@CL=y%M4(chXWUA1dVzjtZ=_38De_rEmV*l(Tg zyu7`CeR}wOX`7!e-+tt2UDNf=#~&N5Nz4Dow&|y~`S`i{yMKRpfByBUdHVJF)6?tY zQ}gxd^TYG&+x738k3ToxeY}k8we7m4+nR1}o6ncAz4qO@Oij18*Rkunc57ZP<8bZT zVd~}^c)LzZJMi#s9GOS?O^PwyE3zBS%U;TtB~mB@|Vkx zm$|!cZ9j~oAMx{L8Lw-*PFn%Lcz9v3by?Bd%Vi#~^E|eQ*xiHsYdelxuSqcWu#U^j zl$T?8-mXjEGXHYydvJdp`msmRj6I>Cab7j$J8-K(ZPu(!@!RWF3fj;F(`+T}dR>QY zMOU8-ctZQjGK>tg4cConYSEYZx@-eF+4}1?t!;};%W@sZcA2JTlLS-03;|r{tsA@M z@v^nob?T=<_pjG}Tc&<)R@5;JOV45ePSL1V{xq&@uTeAjcv)wxY#KMr%;0srV&Uze zIV?!EPN;Up^44vfB*A=T_;x{n6}iT3T>wY^G|j`>fDQPf?1%+h*I}H;0&E#+n#Oeh ze(lU;w|N;SU7H4-r|URu9a6i;V?J%y&&bU4O;i?eTIYU7RGSnlw=P4=vz!79WB^l^ z!A`A=iru11w*q=nhlfc3w_3m(%hfe*Rq@7Y>$-7noMAA5F}SPlH+$jv(}M5L$uPQp zVFl*)y750eo=2_gHjgq^LP9@iH4$rVw}}M+(;rx>U>8d-%WArb3C@eWnyF6F1qIGA zjG&g?y9`7v6V?7iSpm+GV(o^So?J<3_~>+lfc`=VPoaz<}K{0h5)x)?u0nj&4@? zEpd@0bPC8=FpIg>Eef3RLX2XW=2cz+Hg_9B1@#Z|#sKy>?zT$*m_j#i(mzk%mL3fR zvl!Y^Z={ihm?Fsfi1iutGL*-!GwTv@N z8;&U#Kg2^&Bd=VinQ$0v2^;i)ztahJ2R3Ft2N{T03jWNc$CLVH+rv-9 zyB?DtroLx}A^u=3$kbuT3Slt-_1moE_{h8oXG9I9q>zSs%#o|Hg>@X1KM6w1JZ=jY zc+?r2#|cAoH<9AF+D0Z2q4arWL7g0mMA4&=!%3|GvBjl!GgN#Deov8L`Y=>u%;;&A zTXcpK{;vGZ9Q@9WyeZ%l{)3B|90SHOZ*7PjCiQ@u?wQVJ=uq!`oySdkZe*wF-sHe3 zZC#wyZWY}OF`i@zULS_a4gB9^Pn6*-*I}DFtw)ACBw>X%0+2jxC9SfEOl45elzCti zEoLaI8AxYc+mbFjiU3SyA#e-bVyNj9bz;?3px$cS)rw~`ff(m@BP-CiZ98LxP7H;~ zRh}(5oXm=;EV4v$)3d1Tbz!K+=%}Ycf#`y@>~tXTcW5Jw96cKkz!ydZEXlEqGj%0n zbSPm!N<>w_Ue|e%ty3SxloTSiqMQ7}(86jwZrXDrJ54(|{IWQqZ3*4{++v#xWiSTL zZe*R)DpCWc!Wg5Wpq-|zNAbjFltTb!j^lRi1|o?3h{71vsCWaNH&#QrippwL!k3;V z%JP&26e+ksSA(t{&y?R%$Hn@)qoCe%tKvsYBN}6*08(U%ahJ>bSiU9zIa!quyST3( zW~E%>vF0_5lmOu=DS->^u{@P~NHldVITrb6)|u6nMBKKQ8^YUe*hr?D*GQ2^ZPJa? zl{rvp1(YIcQ=K;w-lmkq5ht&TWY>Ni$)sAbWn}f312II!3uSd&v2}~v8d=&x++aAAo{9>U zvs2rJ8GAI_m&R~cv%eZwRg^+eA_0dd_ADu$!^5$KX(47K(xOUOTL8&{eB_`$1S00X zhE)yL7YhwcQtqWZeQ3ZT96A*<$+TUQX`AYC$xtO~f8iBd;*&NGs2m~l^)ikXOOb&g zdXFWwauBR)pzc#xC53fctgy5U>>Y3=a6-vIZsTJ4Slm$&L76<4m#EjPzk0%NmLQQI z7&8V9>+8h82bYHIC|2={E0$Gss|h%d23vk9-$*yIT#iq~e|&kT=5x9{kr$ zJ%%a=_&ICi%jS@xXQA^DoE-(RGr&*6=GtG}v1gmpPO`w+%EVn&(ZlFA!YqkQdt`rk z-{NQLZD64u@2I^O#p@^s3sMReSCvu}7i0s4!`j`SFFn<=ly2Bd#)1m`jTPYUl1{MZ zETh>KPjNUa5CW?jBRyF$6|DtKX*sh6464nS_mr08X6-DL%BlD!YQdh;vK`vcDIk$| z?0kq%kC(om@J4DPsfAtjw9UiB)ues(g76s9#2AC4M~M!pofOB|uf{4jC8;%xv5%}K z04)H@buWj?!eOGK4|15)gyX3VJ1X*Do}nqqsl!0TaN09m>)LlTL{-rgc|`hVZXKfE z*a`uVPR2A(g%wEjepy3elryg(qM!x*F?DvxBl||Pr&gnoKz+EX>ff*{?x}va-P;G& zDRkrtv{0;Ft@?o-4AC}@911kmWR!oyxPc-^+MuxPxq=H)PZLcz`5)EqNLMCIP_>PU z8+wsW5B!uWO<~cR9Z*>_mF>Q%x4%QXq>WyxDYS?+JDhdOk&uGss%4j?cX1c=DA?G7 zhb?0f3_b4So*(}%?&6;098+nFMsqwYaK%G>Eh`{u%K@g56k(_Xwk;M4_IGHD>TkhX zlm`rFa9TyFQKR^mw(KQdUXRoxhJ{1l!=sSQsR)Sl3$ODMI*{5Du~S5?nx4bsoo)&h z;GY>{-AYnUahFxiolgot!(HpMSC?)}9m&1-$^B?p0%01t`AN-0P)=KaBIMrv>}d-2 zE#ZLZ#UoSulN}Vch6AZyp&E((PfFxmUKxr9=m!CyRV8Fs;}fjBlo{m2U1u@&mYATH!yDDA>5G;5xk zp?-quDa3B5SBIgf!zs^juV_Zbli6+fRng=2j;LQl@YoXdt8G0|ZyU@cH$@BFbw$G{ zb3b4mgt*ie)${s+3ST*l!aDXvB^|qAm4bk|F{+ugL~GTx?4gsxQ3kNH(vHmb)zT16 z`WsXRqa8GDIij7r8q?4YT-7_zOTAI8D*ed7i=c|VIoeZ3D<5bFn46?v8qt z3v%DQzZ?Q`su@MuDdmy;Ar%d?s%GkT5q(>HMn$ z6E?Pnzg#@dSeBY-$M-3T-o;&N8Ym+}%`5ru;6(~@T0p-+G;2||R z1;kM806%kuYW7!&YNRvem_{5aZ^?~lEYRFeQB$O*8U3}!G8Ski8F{wn7{gCVN7zXe zquu8jMngTpD8H`x#bZ=SuVxqTpk(WsTO`aMpakx`LU>o|x86ZDdz)&Qz5VoalunE# zVn#W&I+SQwbBg%qEu~0!uNcKZl~jZx532Y?u!>GJ4Aw-2U|um|A#&$5Y3t9EYwD1Sp<)hIMcfdFKsh~%*PmdWs73uLW$oKgkt?S{lMzd(s6u8HQ=2h`8FEBn#*8Nfs%S#6oc)X?TgBWqhI5}84e((+)Vv{2SZ-4vSoJ8&!4$X3=?wT=S40A*b8tfRh5K3-o zK~~w=C*$rggSEXS1&Jp$C+H2zQG!I2nh_+X)P$gd$^joJhp*^B1&fLeG?1pGQDp`@ zF3^vfDY!xcu-ZF`JiP+uRm5@P+`yn4=@o zTQq_!qmD|G8{e{ja`>A3qcp0yzYt6!pT-71QAveyIGDgx5q}EO74N55cR>3oELNf_`=c67wqWOSpmXAGIP4WS<7g2{5-O~0*9G_gIqWDlv&G5;?qIGJ3MDWE( z2~Cvvz4)8*JJEaGbj0pydzfDGGjb16YHknkh}zS(vtsrDYhuqp#5NMiU~x#9TZ-}< ziFfI?$sU5L+417f5m2jrG40aSABG(Ka;1PkxoGbng<<#_D zp-5xzaQ2$KV4-MFmG*fg=nSH79ieciyX*G1 zza>1tXe2WpaYdQ4L*EC~T!9*IWw0aW0Bc?jqdKDHNCq`4M}V(MIo$n-lf&+6N)Csu z89CKx2ZS6>So3kF8>8b4qYG&IGWdp!Gu2DK#G1&(8O$OY6=$$x;>Zg%5vK%jz{7zfB2=PBN;= z{t9gx$7V3vbg+wOju~UEa+q_lk6n&my9c}Qs9~u`JX^XfGljRm{nJ9sGBS+? z%#A?T8dur9cLE*hf=k3!pkrr{DfI#!l>>Ohuko~+UIVPzH3P+!WOB`5M^o{z)Dd)? z7NYJgLC1e7j#b1OW|_Kzjz7bvb6e2Exi9EP&7DCf0FItpgHBlNw5HUwGp`sm14WXE zFAHp>!H!R3RQ#}{)AXdhLFm>CICco#x>}3S*?_8+{e?ddyC$JAX@shpNCT{RG_c$! zbTk^<&s!GFcBGCuGEjDrIW&W{0*ON7Q8k0c^!Hw&@mx$b^$Hy$roC%CT;RdNEHc-I zg;wMl7Fcm-ETg3wlT=&Rc(%aJ39sGBFrE`R#Ei)#w6_ktSw^MMM4N%ttQlZ#8#>ZV zyQO%zz;R}PsdMNSpYe;#}vg{u06NioMt7N*< zJ41&-saE#)BPb% z{obWHk0ly&=H7+zq@ja7YgoL0q)>DoOFWMyQkbg-Nk^IM2i5P~;1v)a} zo%j6O4Yle1USxjlMm0o8xV#t=>bCP(Vi9Z|?RXatFS0n|kvFT<>2l{3S&m$c7|B=+ z=wS9n`TZ89c*m{imLvAj;f{XPlIoS;^H}0}EK!xk0WE2;<0aM0t%%7GA8KauJeDXb z&iKUhSmJps@jRBOBko5`rj9GuRHoiZt+~weSYpj& zW=%tiKaVBuEfnq5jwn>wam}HIPaRbxszVDF)!~jt)yNf(8Xk8*qzbQ?R2{lar!5?9 zrkNaZsgfd}xG*wRXa8z8RcJ-03OwRd&tr)@3#=?xldJOZid{9ZpjQoc{Hnm3U=@1A zuu6!UVij6*tS-RBHq(j{sM>^IRzG!MM9C7|FebCh2 zH{yki_&rnqy3u(oaZiNdJ)rPiR{j*z;CU=DezY{a(BsHKA7qULyo2Ufpi+13JeKG` zQmQu><5@+Tl6HL0;PY4_AM-emC7#C;^;Us>y-{RM{477V5lDsb^-&df)yGF6aL!|i z6sO^PqjvtVU_lIq&^V7JZVer(^H?HZ*it_{eqUVn!mB*-6QpVtRNoY(wy(K6HUUS} z9dOOuX`7Hf2x`QgfrZ5yT+w#+!g0ac8C;NdzHV`x-5uC(8r7@NheQuCw~ch8`hBE> z{YFxwCbsS#d>%{WP4y<2l9ed2J0*gg3FdMxp;U!Q*D zA0S*Wd~hoLN6pFC)|($2!?q^;UGr)f-^;zl_^MQe@p1SH8#stj-b>S?02l@$M?!!MHY@=NU}Qm2S26_-0ePi3dHQ^(s&Ch6 zQS`zPe^ytW@jnA!{Pd^Se{f4VIAhKIyHAJjzB3nZ|NHNwx%lDoba`|C(Omp=`SJK@ zJO_R@-+zBN-o2ND^7Hzm z0Ufyg(WLd%*-rBFS(?8d?>?S>H6`RGA08f0568>fgEb!yFW>T*GtQg0A2655{2 zCu`o`n&18X@^t;>XpUd5-yiR;j^^QbdwG5LxcuI{{bXLfJ;b_L=Zb5_WovE^(JtQA zl8mc%iOxIQ%*`Q&#o3Tt{t9kODmL)A%VF_8r|dbM@?w)M6|wXssV?mscPF>F=u@d( z0^Zt5baVLi@Zpf%(yR|L>KktlB`($0)OPTVh6|U~Qkz4p9G9G=zqj^P&WI zhZ;*pj+OA@th9ywbinMj9 zp;c^kyMq(fFC|1SYGG-pDNZkWDJ@`==9iYLwa8S;5~D3CnI;k>UqT0KZZ0}=b!c{} z$)_OS*TuJzd^Qzxgit)r08AsOD*uYLdbxE2uMRa6WhpkoY{E4z(Kj1Jhk{f!VcJUM z)fSUTkQc7E1pz8@#a0Vo{9R?ibY*7=Ogygx<7fvIMn5p^ns=V}R-x0C@U$OY3_6IN#BR^k?0Di6p@jtmn4 zl(&Ez?s7(Xm26DSxtNU#3>1hxUFG{3FT9@$xtrI(aK7MyY?sFW$atQrwdE+m3MKd; z-bAcwTfzZg@(*0qC!MOPZ!>2gM^5S`OtQGub(maxQ}ODjg* zN(|b>6Z(CE6$b?376qoPd{slrkfYLyyoDEWp)$aR^{~iRJ`v&^2MLOja+RzA%dR0* zH$TO|4dU}8ZmVK{Lc!%G_UH9hJQnC-5p0uOZA&DFOVd$-#w%T1b*gP-nTtK)cyFx$ zw+H74sp=tnO~E%ioU1mnb(VWVY9ywl_&4!RYSfh_W$3WS5;3R&{iPD@8FCi&9N^fPeh3Hgw`N5jrPL<=%G0$C_*5Rh+KisKb}Luastl%!sU zJmJV`Vxh($^%D{*Ikv(9jjCYNI5d=9qQr4}-V708^jvYMl0%qiWE65Jsf8fgaJAA< z_@(DPO~SOtMCFc*ovNgTn>h5h@SEK8PTkyu@S*>nVpEPlu;gYtcTA}RX?m`7Q$vO7 zAhtOZzDYNjd$Zq*h*38WKEfBxrj~RMJ5IX3IngoU(+LocLpd9g^Uo1_4Qi zRmv((M3o?Znnth0iiXOmeH6 zh3KudnIKeR2u)VWY>`9BES!pyMI@!(WMyrOiOL<0eyTT!imWA01H|vJM!$JvHW@%J zL?JAZqr^;K$sGMz#YA^eqw#4yb+fXxl0&;i2|Zn? z^i$5BDYSuuI9%KB+N8;mt%gkuPcC@ zc%^d}-}#WGq4WLie8Z> zkKV+^w4ei>R!3=~n)G=QdP{2&N2Gxwe}qfcUXnXjT;x-@w8<`r<%&!$t7ZzoY1>pI zFC~wE&|!~f`dgBP+BNa+$Igr>pl8VPl#Czxfhe-26TRl8h+Zbyk(h9u6br^kZ zDx{rF)^&|Fy2Y2=p-wYj#VQ@8pcoOt;YE*2$s8Uh7E*z;5vfQgEF*x(fqc{;--!s_ zYxio8zR_q9l9FChr@tA%oeq_WQF0yEB($gLNy*+N%6y>>yT_9Z4wxJvxAPJll}M2Q zLv&9hSxFG0D$w@{ty02V8y%Lof!Tpn0*6WhwT;7)V`HL%K~bLLdsN$1pHAdW=@RLS z31iUiUnK@1-woUCs)R1mLQxiYQ7O0?YG;lO-AD8^>)N}J(0Lt;XAEL;_~|@RO&pUe zoV#Yfg@0*|DcP9oIb6-Ra*y<ACfYw}759&+H2yGlVEd+kLhhJ+ z;HRsD_lazzCla$TRi~Ci;xuJnc0pteWg-r8)F|w*u9LM^GO3ZakcvNvZZV5M)JJ-_G;sE74P z``&~K0gf#w42+c-3PreW4Z1_sEU9rJ;;{(THw$TxRj3vNUf6b_tP3}x@%g$8)e6c^ zVQfRSIVOrZl<_964b9DDGSi0kiadATh-x2#kCmwQ_S}i;*g%tQDZ0beRE3y7-HnP@*+=%ji(rcLa@5mCOFfVYa9fUgD(#m4+jdl=9N6~e zJDf+xxT`KE7RG*5%Yf*!ZYhbWa_fdvYbOTiD^hKW4U|~HYKfKg=*-S%cOb;NDD7^pHtkoXIw;80-LBNrJNB){t_0m4*!;@|rVqAA zz73oj=dM|rSjYRdh(3#}(iCVTd(SKKKZEZ|kh=d_T$Lb^lRl&m#stt-ns#_EP09d+ zsuiF$7ple(=0Bz{gxSKHzz4kom^bn_&&TmM&GoOuzdaoJz~lpato1*W6V06YU?$c~ z|DCy;80q*mM%(fPqj7kG1^%qCeS3PiyuLX;^uKGrx&GhLWd6i^e}C$KW9@&>emq@1 SoVuJ8wS0K-;?<9@5B~?4?n#RP literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/segment-cap.pdf b/benchmark/eval/vector_routing/segment-cap.pdf new file mode 100644 index 0000000000000000000000000000000000000000..13aca6057415f13f7a982aeaa3dbf28cc8d591a8 GIT binary patch literal 5888 zcmZXYO>Z366-4**D|#a!S;Xw_`34LFks~2MK5SS762Qn}M7ER>q+vjg1Sh{f_f*xD z5;>b&uQ)t>-O+Kso@YP);pOk9Z}zKc7>9?~?^fS_JKWs<`(Mj&^Zn_1djIftxcTw) zc6l1+`oK@aci*j+%RL`t{r|t<&)xj#eE&53yy4^jD9Xdf%XOIk=NH$%`o)(I^^u!b zA75Yfhxzl1~T;6|LuIIbcaP#7F_i%r{)UN(^zWo03bpBj?*%^QO*E{1n z9+&zl%vwG;H&!zE7-sH-BC(Z?FFxw)1h%FRq@hkIU)9Y8>9Kp5N9Vn}%r~ zZr{}HVo%es8Hag04!0kM-~9D-J%3t;<c>!>l*`{#=hdQFL?}es=MkBPoYjzu+tpwGzUD*OCS#czEOW?)CoEHgWe&Y(EK`GJ4o$`~ zHCX1*WGqvIWe!coGBsG{&}1xg6S6F$k!21c%Q7`s<`A+hQ-fs=AlD`kJea5k7Z0O(+m&GSYw%Hcvz-& zcv!|7%QVBoGS*n88J;YokYx^i$ylZa%N#CS#czEOW?) zWtp1l{Ue@bYOu_q$vn&4G#Sg(@GNs^GM1^~S?17WEK|d?%%RCV%V=boL&$oTso`1X z5VD?SYIv48gsf+onZkFd<4$ylZa%N&}FWood@ zp~+aL2Fo0pjAd%D%%RCxMkC7{LY8GTShiypSF^C1MP*`{W-&Dj$;2|v0&3PynOLS7 z9+qhx9+t7jGR^R?j5Q}l&G4{{H77>R@HjED#xl+DI5D!uGR^QfF|w9r4t>R(7~O=a z6QdeVj1Ix-#HfZ7qeGLiObsVShbD7kRD)#>P3FX?hG&^Wld((%hX_* zLzA&g4VF1H8OzjQnM0GYj7F9@ge=R{V3|Y6vP=z@IfN|B+%y@>)L@xI_AJZPV3|Yq zEX&kjnM3wGb7EA3We!c|#Ha?#9Gc9DQ4N+kG?^2ln$1!< zyxOSNWtnGqbz;=IEb|PnPK;iMSC(mACq~ck$}+9%#ON6wmZ{*xn9!G8Cq_4QVoV5? z6Qi3tF(w4eiP242meAxnF{Wv9S>~orj0sIH%iPrQBcaK)bW`tT2~Do|GB;&eLX+#g z%uQLA(ByhAb5oWjG`Zf((u9m~VrO|JJc zH+5o6XmY)mxv3LlLX+#6lBUUJnVUK>CN#M$b5keAgeI3|ZtBFC(BwKXx+%*Nnp`JF zHCUDqGABkgulQ~=-(BY0;9nk>H}yT{ZZ&@X->}%TW|hA-W5&`*cUm#&QYx|`~0 z@O3xWKjN>ub^X)&++80}=l9E_zQ2BT{?9V(>Z{y`2fljF_tsC>)8ke9-0sKC>e;gw IKfGN17pTpc2LJ#7 literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/shape-cap.pdf b/benchmark/eval/vector_routing/shape-cap.pdf new file mode 100644 index 0000000000000000000000000000000000000000..09d0cb0ec1195a7bdb6786114c6174a20c2d43ad GIT binary patch literal 11699 zcmZXaQEwaP6-3|pEB3`e`;d^|E+xxA5Fk!%1W4KhiGUUe@<2;#mxfIh5^bdY^}TcL z*fS^g3x{(#T;1d4c*TA6)8C$dF}{4f8PjgMd+~bn)mPK@`oI5Kr|q|g^Wn|it7-ew z;nn&u%{B1z^!3-9^>nL2>3qS5x$XDI+lT4by~cm5l)HDQ^EB**%b#BO{H{i}zr1^K z?l;!kk7@h#@UVVxzFXhCU(d&z!?b;Ny1BbOp6aOnc|3h{dN_W1;`5Q|t^a&v`Z?~` zdZQ0N$oB8+?eXyR?)T}}UCr<3-L&j>b-+Kb4|nhGZ|W!3D*6EW!I*U5+Q78^*N-n= zuQz8+Nb9L@>u0F+hw0+u)E|42{^MwVSWmCc|DMJeyGy-t^Kiaj4{tZS>DA`xb-mk| z#ynlWtbOUA#{DKaoadC6A(xC=GHS`FeMapwst(?d>LQ~qGU_s;E;H(JMm^4`#~F2% zQCAuDB%_{W)RT-FyUY?h|CtIVbjFQ=JS>0A$iw;v_CEk%fWQEP0R{sQ1}F?*7~n7f zVt~W|iUAe_Fa~H0;27XBgU1XWGkDB66=v|5!D9xG89Zk2n89NPj~P64Li@j;89Zk2 zn89NPj~P5>@R-4429Fs$X7HH7V+M}}JQna+z+(ZA1w0n;SioZej|Dsy@L0fO0gnYd z7VucWV*!r^JQna+z+(ZA1w0n;SioZej|DvT;IRh};n-(UxU3H=lVahrKB`QLgv|0;U1pS)Z?1N4Qmc2FQChsYT@;}+tc$|^dU3p7 zCxZJ}eiw!NSbrBq_{al+E()IqgD#5jkp~Q26h03ST@>LX4=lPUd>%BqD8kRYE(-Iz zC|u)A=%NT8dEn7S;q&0pMG-#oz@v-8=fR_kB7Ee5M;C?9gGU!d_{al~E()Iqk1mSv zGp~!n{4NUD;L$}9KJvh$i^Au@ql+SZWjs+*eVAMjn>mMd9;Ue-}mg$OC~c3ZDmq zE{gDx2Mk>lJ`WIG6yYNeEV?Ls9yGcr!q2=e3iG=tT;ojWq6i;(;L%0l^Wf1%5kB(3 zql?1l!J~^JeB^;g7lqG*M;Ar-$ODfq3ZDm$E{gCouZzO`E(+J+(M1tH^1!2u!so%G ziz0mFfkzjG&x1!7Mfk`Ak1h(I2ahg_@R0`|T@*eK9$ggSXI>YD`CSyI3rB6(fAtiy zCe77jmzN=ngu8WV7c!#?gS%A}E(=1)jH;q=w+MvHsM^1~RTLpJs+R9=6@|-MeaMWe zqHtG5kvhXv6yfXgpIcCH1e>16-CU~rFCc!P*H>r1}cj5kp~PaikJro z6-D~U0}B;J%!7uCBK^!$QMj+72n`-8iu92u(|D*TLW75jB7Nk6hl(QR!9zunKJvgr zMG^Dhp`u71dElX9{JX93vBM&@O6fqATDvIfaszKA9-NWMd9@|QTRM~bWwzlJn-nE@Okj)q6i;(;L%0l^Wf1%5kB(3ql?1l z!J~^J{LJg3Fu#k!HF$JUgpWM%=%Vm>@aUolA9>)>Md9<{(M1tH^1!2u!so%Giz0mF zfkzjG&x1!7MfjQ5MPYsyh3Uef4O&rzyb8BixLcnqF6(i~eaeg~3_e_+QZ5TZ$c(C@ zaJLAA%&6MGyVcVnGpd&FZWV>gT7Afjs-kdLMUgtgbVU(<6-8>~VfiYGm@knUc~unQ zgMh9m!UqEtMf%7C1{Foj1B8kqedK|KiX!GgLq(B(=BX&$S5brp4;4lF$O8`*Ma-8- zjXdzsm)S879{MsneDKhh+2MnSiXwdQP*J3hJn&Fa#5{PYDALb76@~jMiqPPpqDUWk z;Gv?3`4Xv-2Ojz|JLbVdUuK679{MsneDF|FgbyAniu92O9x95M2M-lR`kAMqa9>5? zS~%*zvR)J}E6k)=xU49X6&tb+%-xI<26IKhq#(G=C?POc1YBm6_HVA%?=qvbd~>yX zlUlvYj8ajU+eHyN!@4Nk*NG4sd02iIh0kOCx}u0Yod`Y;0$mjDgFzQX_{al>E()Iq zh%Sookp~uC6h03cT@>MGUKfS=T@!L8fi^6o>wfymB+mH7z z|9-!|tQ&7HH@i=Nrpt?q|zVEpH<@mpKx~!Xz@9xfh^Kaj2{ct|qpZhrL4aR=+=+UzupKtyT^6`;U literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/sheared-form.pdf b/benchmark/eval/vector_routing/sheared-form.pdf new file mode 100644 index 0000000000000000000000000000000000000000..f9766cb93bb5c569000353e3545f7449d272e815 GIT binary patch literal 7728 zcmZ{p$!;CT6^8fy6txkMEZnSl01N?9EFwS#J2HU;FtVViE13c>nUIv=%nro zQD<*19}ex8KOJtq9v-i+F4fuF+pD_|*SB!>kL%k%+&)})U~uNA1O9Tj{q*>&iaCdq zT>t#>{pZ8geBk-r{gLzD-2JA0(R}QJR>o?$e|LDe`*MGU_~_;JYmwsN@&0gmT=J@o zNv0^T)p60QH8}+p_3Di^x~QAQtwy_G;tnipNZJ{zyj!h}AzI!van(VmY<91jyq0gN z8nYU2L&|dS&5n(m<=5roV$E8#b>2_Jj*Iha);W|Jg~g5rdTs}M&upc7f`!8>n%EkZa^g-vnl^PD#8y@t1ialYzEOuKu(0y zaGBTaN=7@!84aj9CFda;+*%M0E<#*WaiRvhmXNj1Dx}qWokCDS)EsPbg5_9@H|lB$ zddp%Q}MKHlE+NLXU>LEdZxeOSFK z24=^>u%(TSu*%7uMQjyvv=J7&D3Hm?X^!z-=z;NdSIJFSQW064H)g_6a<~6@6ZQERQ%NGiMiDR5fxoAd zaYoeba7*~^JcdGag(&H*%R(%Y*|63SxthJ%fw&Z%tX}|{8u$qO7s{naRk{dZy%S0+hj|Up3s{K zg-TDA5ehC8V(odf=A{V1NH!B+!NJDk{LSFyd312t(tP%7a1J(Wu5jcwOG?YchD~M5 zAPvp#Br%N&>1G#Fi+v>uN~^D3MQc6rE{{m#KuIvv>TCx>NKe`5Bvh zXuL@di6&5k)S!q%pw*b|bhads78+N};1YL2qhiA$jiiPew`#~eFusmxBa43zH0PBJ z6(w{A#Ufdm6fIHG&n{R$7A-N~xhw?PR)D%R1Dj)KLc4|r#i7aE&&JkW=Yz{(ogodr zKrs>94!H*=UOaF-qy-vokXBcMLD?PTbd+Dn6%wSBH?LsaNmj$$7+K>@_&}ky7_F&W zfucxhC#8awmJ=j9wEkwoCLb86WSmHbEb{e6QIfCLI)>?JNnu%nkh2GmlI*Y?Tntf0 zQXDc5jIRxCOYOY|ik~~uu%RFc#Ttb8DQ@gZCUSQ2k!}s&M_Ol`Ax}w5lVieDX+m0A z5NRaq;L#nMePBYnh6cr<$#M3i^1);`&5&gnum^mHS{5@J>oxOGOSks&xEPJ^gkq(V zixx7QB1bPQ(m02afJ$oG@j0h2EgdB8d@j%TR=LLLehr}G2Uus2Vq+i`mH$| z?+RlBmNZ^ORY_X?D$aPsJbea`BZ4&&V|zH#C*t>cf~&z=Cn5FBdQb5Z+a!;j$lhYG z+L$)mXy;7(%_rF_uy1g;eM+2RiOEUcmSdiie^U*ohLV`TC4SL~G!DAje5VttL2R?u4#tJr46>W! ziN#30VRqcT%tIaTYdIddK^iK_wv_{z-*L2YNwZi}(cFx}EPW?B-XBt-CieL1>GS-F zOvWbR%d&c?!~O`u0i*AlE@d-ds9&ua+@bVtkwHmR*&~^SWL#`26Lj?nafuXMfiwRi~k-uSRE&v!1MS26vPh5h! zPeF`fHn%9vrYvQQxG6qGX)^1)hzc=43t)#)>egun zcbe?)N_}SnAYV4BwNVnYS+XRRNjPX-J_hc6ROL1VR!!1!Wh#p(9kHb8pf0dk8?y5x zy6Y-x-a4MY@m6}tZ6+N*9T!W!x1x<%ZLEg|yh~h#1_1-Z;Y;w8yDXWwu8!&&sHe0E z-leZW`z3fvU&5eoZ6zw~S#5@ERvS_`1&!ySg_bX9e+ime%*RhCh)yiw6r}J0qacer zY+DMcb9Ip2!TY!2*5f|=jV3{Of-85{?$eqzZjGrf*9a|!EJJ54ySomz&j#mDpJyiO zhLe0?TjIokeVu0~c-rS#{MeSZYdbCUXo~S8VMRutZKFrSMN(};@wC#z+i4-BhZA?% zc6zKSDUifQ8^7C9&x*shriVATP2bhRzKGF7npPpO!SuUp+Jd_8^+-pjagXRdE%unB zC6c8Zlswi}dn8u0%7iuV_GWMz$|hJcY1{5iR3+!O{ocd?%IFKeiCN^{Hhhy)O^|~q z9@v(C)9D0DFE!#T8>ce1wzc0R6H(g+!Abm%iA$iTz}B_)N~bsF&O~$~X=>a3Ro&;v zI!VC3{Hu0~v>o&IuNpM#u-pGl#vkdTjn0-fuC4;+!a=%>bx|SHB1iUyZGs`x`(ttj zd81amm*Bac(3{n^^GQ7!@LV`1+AqO#;ec=21R0p{lTlA5SnA0nbxXC+X`$5R<{8_$UV=*sP)Iy+s@ z)gO*ZF7w{0cK0T?iMNlO^PuAP{^>SSo=<#jcYcy^rwco+J}qx9>R~jy7azHkl)F9_ z0$N!v{4S2_cYnKly#9Jnhp*Qk4!2hab$>WsUf({fzgHKZ)!U2ZKZ>?r5ku-b`HwvC zC2@Vl{o(mh{cxxLE*$N3_S^m8K2SyHhiqUa>gVFjt z#h60Ti2NJ$l8Y0)$NS6co5Ov*@BR7ue+QMg?Yz5_tJ>|J_u=vK{;~4THI`pqy?XoO G`SO2cWKmyPNrEbDk>$je4H;z$X%doo`I{kNC(!zC%GTgd?&vO&oJ3U@>CldvRkFGQqzKa8JHNwlHC$X z24jsl)hM?3^wS2nC(zzi&IgD+=vl|aI3G$pV<{62l7}65X=usvGo{}SvNYwMOB}Habg%G7ES;QS z#;{Xz-tl7g<0F=&{J#fYXsp{%({1g8`$x$51M6BAZrIi~*d}y3(|oK(quK3Ty#k6y B!s-A3 literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/stroked-letterforms.pdf b/benchmark/eval/vector_routing/stroked-letterforms.pdf new file mode 100644 index 0000000000000000000000000000000000000000..08820429763eadbe8236d5200a79261845480300 GIT binary patch literal 7939 zcmZ{p$!;CT6^8fsDQY7iS)^I>02l@$M?!!MHY@=NU}Qm2S26_-0ePi3dHQ^(s&Ch6 zQS`zPe^ytW@jnA!{Pd^Se{f4VIAhKIyHAJjzB3nZ|NHNwx%lDoba`|C(Omp=`SJK@ zJO_R@-+zBN-o2ND^7Hzm z0Ufyg(WLd%*-rBFS(?8d?>?S>H6_FfgEb!yFW>T*GtQg0A2655{2 zCu`o`n&18X@^t;>XpUd5-yiR;j^^QbdwG5LxcuI{{bXLfJ=mp~{~TgntaHUR!{Tg6E`J5LB^4VG!R4@cpHudnPI}hZ2`+Yic|AM#F{6YN^d3R*p-~(ITRI4)%+U zv3XH~yF-m7qjLE+oZC{oMgRTX&%u5PK6(tz-3bH5Tt($)U@MnSt?|w7XKhz8P{R_a zX+_$))X*xny4}GE>lXrvi&|J3YKqfKUP=qtr1_<#YArIAvczahN~Vbf$(PWFs>;7&tzK^3z^g;eL|KZBFq?3VOZ3eK(V-w! zO_;V4d9}qP66A&JZ9#yFT(Q*x82MMqp&GD2LP{Lruv$aRaR*v%N-3`2pU-BLZMnoG zXV*(}T4HDpsnzp@k99sHGw)APabT+1XGEP!;kjCZC!MORuDipyvfqH`K7DlpX95|)^A zX~n2pi9wrqLcdS2;($QhqQI1uuWCpca#UK8x9}n^R0i0v9v0ckCqkU#AVE=5u96jC z*)@df<`0sM9rPq_t73mb!R03Q=k-=R7U*FSY?EAVOC*O&(@}xOD_vZ5s%>PMi#_3Z zZ><2g2j>W>>LGhg!8bdct2VNAmU}{KB&MYJH}Ott)RiS==&;8UF{lCkr4sBJc*;Af zhVyYrAs}>@0lpIQM;A{f`O?nxGjQh#`H;Lv!_GfM3oCXRwBp zq+W(R;mB!Xp~fKf6A~&pw!#68s$kPNG?ZPU#BqAw3=v`UTydz9Lzrk}6mlr3g&^8+ zwbD@drRP1Z!nDUk<&KP{@e?Sf@W4C8;yEu$$z?M0Zi6@o7DEv$C|3L%T%@ zJzc2uQ_h|#w1I;-UBu6;7wG}l#+aa?p-riI3=cPB90(X4V_RH+K_ri8j4_QgZ=kv1 z4XIUhR#ghV*eTJLuT?;pf&+3Iv^w6DyrqvD{kNl`KG#;^k5NYCjz|L}%rtnHV}7)+ zD}bDMrE?eG`H-dL!pEW)^GD-BURevcpq|K+b`Ob?v(#AR=YV)uB;vDkyCJxBp;1gl zuSk$@+n-}WEaG8MJAV3GX>zZ zZK{!%lE**ju*Wm~Ey+Udnt1o)=yZB@^)XUO#bYJnJ#>IWqWE~ zjJ`G%(#|I9y2cva;!EyOr-wfbRhswk#xsGcR+SBx;WbYDXzR-r<<4FbwOpcJ-c?ph6 zq)31vx+jvXBnVLz==+3LDPgXS4olp?>_94kLnVRQ#$m~^F;T&wD9`ads_m*zC-SCr ziS)&UF=+R%5`&QMhV6D$LKkVFC=0x(6xO&J)$d zF}cFIYxZ0Cm*$v~jk%t~)qE@WNPjA}-DcqpD{4B1fTe+mH%{?_>8-{my z&tIkbpehAuon`Q4IHc(*Y#w^DV<092@}v*f`mkeW%j+b$!x?49RJGe-xCYHq$Ye&= z$Gs)bWVbZD7Nbw>X2ZG(^oLIX{gv2+ zXx?O;am6bN#{->MWijHZifhxlgJ~@@BVae}cHGliQk!M6kWx;^m*@pOt)&gNZwe^n zj>!jpx;l8D$VPf1F$+_5YB?lMQ}$&SM8;4i;vh$j!Vc>?Y20x>O;k21QnlZ4zFC$4 ztN>`&wH(qGhD1l-<(N`K$I14Iiu$){I1Q!DF+s#o)|;d`PZs1{l^-z_iH9F!TYfjYeVq4r|f<6{1Dz zz{I(nRnfYr5&m08_C3BokJORCBB7tdhmh#BBSiLvOD=r_DI*b+B4!o!6h6LGDm(%7 zo1N<_MS102yt-{ZX#gj#R-azBbZy;8o_C*okA+1bLStKg(lbHI>*x-v)kidZ9oq+g*M8T(IKbC>Kmb-cp?R82E@ukZR-m5wjW)ndR4+b)!K;U+XbUzed; zLD?ycZKyWKL@|dl-o&+`xtUC6+R$E+=gu2Z?L+Xf64lYS!~lIos!g$h5-V6Ov9cbW+4-#eg-Q(6 zX=z=Pc5l?oTCJnop0xWe)7#(3pxXnRf7!tF z!4}E4fm7q$HA@rgc)u3WXK_`U0&QgPc}4zb@LdT~_dkoP5+richxEai0NP5^4)3K& z8DLPg0<`8r)%+Rwas2sm{e$pt562JuDKj5fR;~Y;oaorh2Q#r|`tQu$#K^g?F?y3H z7~R1WEbzC3?c3AC<@L?+q5lc{&Gr9|CiBb87$7_2cRC;nd|U^tp!@FJArl G`tW~UG(AoZ)Uj{7I_Oj_K9|xFSOcH4z-7##JX)lWLFO% zeu^o)0!L>eed)7g7eOH3-oRO=i{p2Bhxb5Y;--@n&$M|0L1Df4^r`=6Dv Q?{e>Rcwy9FFu9qk7g9{J9smFU literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/thin-stroked-rule.pdf b/benchmark/eval/vector_routing/thin-stroked-rule.pdf new file mode 100644 index 0000000000000000000000000000000000000000..ad904c9441ad6a65c7f940542bafbf1cb0155169 GIT binary patch literal 682 zcmZWnO>Y`86ukRa%mt}E6x+LNTBImaNkNtRktnRxN>LAH_d$)ouGWU8`StZ%Af&}7 zzwtcJ%(Dk~A7*d#@U75L2#dA2x`M3#{36I3o9rg6An)voB8p77$Mv-!-!hTwe;W=w z`N_2r4;9mY#$<1f-tMn7&irYj?9oel=ScyrUr8IPbdYOmCjx;KT z!YI^Ry!vXB+Y@N-TJjB{qpfv54qu~QV_FL}H2BR8+8At6jt;vG=4|6AbbPqSj+R+; zB-Hc5$e7Y#i8D4ve+qxb%)1Lr>l}~Q3to{^{Ew9;Kgx-hI_oyn^#zKKUk~#ufAm~DtuN@h>QY*!UJkb(cCf6&gI{pI?@gZeqh@qa_*?#u0?GSi3dn-9)SuxK3sQTaOS5A{(89m^!SU4Ifs*6 z|N7zm=fl-};rZSDk^A1<{i=S}eC&c&#%j2KcX+t_a({*R=;ifmk>cU;{&0C*@~VwV zrYNt~anY+aIRzE<>WwwJsGG&DM!R6*4lHX(+8L|7Tdj>DT27j{>Y!6LyQ3zr;{2L*4rNARv7-Tba>~U()A}_=uVL*4)UMk5 zVuf`mxa?DmDm;t&vL>q$c}MpJ)UM8Yi%y{%P>IKE%0G{a@I%z9i}743Ss$Mbg0g(&S@M=B*@wM=x0nrB?P~Hyq)w&SYq-)PBwx*tlkv^ zv*Tje(#A$u<>t;JwhB4g2#Z}5$mHZS*Z3~R}ns zf@zEz_Ji9{jPaO6;lmULcT>7SZo;qO>9v!Ff@JgyQi&v~h%C+7y$3N934?Rm83r3k@DHWOdL!N$+|o59QT=-{xW`Rv!=9BkHH;mB>4l$MDNo643! z8k+HA7pz4&l0gjCRSEZ$1XMGKEm3OM)shHMBA?_aI@1;}Qwc+7@Ay0FPSKa-XKeDJ zagrPoO`ry;K@o>Qt1;W@Zb>37G_IDxCGLbq#fC!~Newk_)sTB&d>zq77XKb-&MO%z zO6UxVMY1v}TB4-iU9f&ET4KI)SqQSN0Ci~wHpk9{b`1@RLzB7Rjjg-R2baS-LmGU6 zVj{L3at}hBYspew(FjIcZOg0%pT?L>lc;>tGE{ z=~mAqfSMNy0Z)Y?bJVmahiq#F+i&4}g_m zG-VC8`McN;R`}*r)dkkZ6)G6gcseH<0HecdXr1V90qIZ*Ngp!Ac&nKmgl$dex8`h| z6~+cEX}pN4lC=6&obeO$^cg^|2-Zl9?cqqDh~MW4t_EwJgw!+ZJ;h6GlRS1JdyBzp zW7=$!OC1BmkrL6Kv5h&^ZZb^JrV2TM-GhX{ILjpBvovS|nwljf*_&*noe+zMME6G} zm3~sXXYmdgqZU}n9tfPP`2hGG| z!$CT3R4i<4r^N*_{(uj`vc+4pxIUia=%vVjR1;HL^CDWO>86hmnZ2XRa8yF4P9G#Z zBjD78pgA@K~VG-1%B zrq3Km7v=(;apjTWL@LPNNv*maI&VPpnFlS?5JmhyhvvJB(7dPBXaE zWIrqQoe6+^*{Ie=Nz7)+l2j()pmq5exc5<&+YneaNz0X~ETVM8lBR>Yz-Dd8&M(nj zS5foU@%tNZrI*}h(($L`V#)Vbv@xrV_0WKKiL1~cU|=|W37&G7B@@@xQC$P|ls3V; z^fhR|1W)Np81${JM1?)8&2Y_XL+YlW@q1{Y_$e{b{)MxX*s0Nf4gk%AK|Qv}TQ4W2(zFLW?2G&{@mwuEXuK!THnQGZS^g zNj|VGabm!}&a)Fd?ei>tY)jja}8Jvc)36@OSwtEv*$+>O6H!*-R`hss_7P+?#-z1il&bH#4 zR4_nPFw`~RNwM|MvgMm>AUB=Xe3QL^+n#T->sa(nK7asy(>KG@s;`oW*3~86w(FCt ztfRG_mwm#PVV793zU>nMG|86cQreRiNEQL>Py@7W`$Q@1mSxfzp!kIhqHLd7%8j$Q zfXuM6y`^ht3qLU+yEZA7h=1DnS;xRz_n>gMZTzg`sx6}X%5MTGntkUt!_aRN6JHgURjrhvOsf?{{?KjCp)V4uz62D{O66h(gb*;V9>7=}wh)yI;ZM(m!`y5#( z3D}o^)lQMNW8VH%gJvCe`@hNfBVDx7+0w?d6F4J(;9#srET7w0go@-}c_M zz52d2>H*MmBIz5Z~xy*jA-!}0R^_F?_Ky7;W#UM&Arbp7o!{?A85^c63M z=STJZo%)CHve(wH_lJ)Q#}>9;+kYwsdQSCG4Xmj8H+4HOvd>eDUh5l-rspZf{E05) zzdFZWyE}QL-ClSf9xv}7EAOn%^787{+aJ!C{{zyM BC7=KR literal 0 HcmV?d00001 diff --git a/benchmark/eval/vector_routing/truth/border.txt b/benchmark/eval/vector_routing/truth/border.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/border.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/chart-labels.txt b/benchmark/eval/vector_routing/truth/chart-labels.txt new file mode 100644 index 0000000..bdca047 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/chart-labels.txt @@ -0,0 +1,3 @@ +Chart caption. +VECTOR +EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/clipped-out.txt b/benchmark/eval/vector_routing/truth/clipped-out.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/clipped-out.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/compound-fill.txt b/benchmark/eval/vector_routing/truth/compound-fill.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/compound-fill.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/dense-graphics.txt b/benchmark/eval/vector_routing/truth/dense-graphics.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/dense-graphics.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/filled-box.txt b/benchmark/eval/vector_routing/truth/filled-box.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/filled-box.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/fused-panel.txt b/benchmark/eval/vector_routing/truth/fused-panel.txt new file mode 100644 index 0000000..e7bd212 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/fused-panel.txt @@ -0,0 +1,3 @@ +Native evidence remains. +VECTOR EVIDENCE +Panel text. diff --git a/benchmark/eval/vector_routing/truth/image-overlap.txt b/benchmark/eval/vector_routing/truth/image-overlap.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/image-overlap.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/mixed-native-outlines.txt b/benchmark/eval/vector_routing/truth/mixed-native-outlines.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/mixed-native-outlines.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/native-highlight.txt b/benchmark/eval/vector_routing/truth/native-highlight.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/native-highlight.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/outlined-text.txt b/benchmark/eval/vector_routing/truth/outlined-text.txt new file mode 100644 index 0000000..aeb0363 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/outlined-text.txt @@ -0,0 +1 @@ +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/partial-native-clip.txt b/benchmark/eval/vector_routing/truth/partial-native-clip.txt new file mode 100644 index 0000000..d357965 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/partial-native-clip.txt @@ -0,0 +1,2 @@ +AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/quarter-turn-outlines.txt b/benchmark/eval/vector_routing/truth/quarter-turn-outlines.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/quarter-turn-outlines.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/repeated-multipage.txt b/benchmark/eval/vector_routing/truth/repeated-multipage.txt new file mode 100644 index 0000000..89957bb --- /dev/null +++ b/benchmark/eval/vector_routing/truth/repeated-multipage.txt @@ -0,0 +1,6 @@ +Native evidence remains. +VECTOR EVIDENCE +VECTOR EVIDENCE +Native evidence remains. +VECTOR EVIDENCE +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/rule-through-outlines.txt b/benchmark/eval/vector_routing/truth/rule-through-outlines.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/rule-through-outlines.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/segment-cap.txt b/benchmark/eval/vector_routing/truth/segment-cap.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/segment-cap.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/shape-cap.txt b/benchmark/eval/vector_routing/truth/shape-cap.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/shape-cap.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/sheared-form.txt b/benchmark/eval/vector_routing/truth/sheared-form.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/sheared-form.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/single-logo.txt b/benchmark/eval/vector_routing/truth/single-logo.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/single-logo.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/stroked-letterforms.txt b/benchmark/eval/vector_routing/truth/stroked-letterforms.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/stroked-letterforms.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/benchmark/eval/vector_routing/truth/thin-filled-rule.txt b/benchmark/eval/vector_routing/truth/thin-filled-rule.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/thin-filled-rule.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/thin-stroked-rule.txt b/benchmark/eval/vector_routing/truth/thin-stroked-rule.txt new file mode 100644 index 0000000..6e5c1d6 --- /dev/null +++ b/benchmark/eval/vector_routing/truth/thin-stroked-rule.txt @@ -0,0 +1 @@ +Native evidence remains. diff --git a/benchmark/eval/vector_routing/truth/transformed-form.txt b/benchmark/eval/vector_routing/truth/transformed-form.txt new file mode 100644 index 0000000..a86f0bc --- /dev/null +++ b/benchmark/eval/vector_routing/truth/transformed-form.txt @@ -0,0 +1,2 @@ +Native evidence remains. +VECTOR EVIDENCE diff --git a/docs/host-adapters.md b/docs/host-adapters.md index cf0cd7b..fcd2aed 100644 --- a/docs/host-adapters.md +++ b/docs/host-adapters.md @@ -122,14 +122,20 @@ projection is the entire source. `locateSourceRange` validates the exact source-text hash and UTF-16 range, then returns intersecting span/block/page evidence and synthetic gaps. Span bboxes -remain PDF-space boxes, not precise character quads or viewer pixels. Apply a -page-specific rotation/crop/display transform in the host. Synthetic whitespace +remain raw PDF user-space boxes with span precision. Schema `0.18.x` exposes +the page's rotation, effective CropBox and UserUnit through +`getPageGeometry(document, pageIndex)`, which returns geometry or null. +`mapPdfBBoxToDisplay(bbox, geometry, {scale})` and +`mapDisplayBBoxToPdf(bbox, geometry, {scale})` map all four corners through the +declared forward/inverse matrices. `scale` must be finite and positive; it +converts physical display points to the viewer's units. Null geometry withholds +a mapping. These helpers do not supply character quads. Synthetic whitespace has null span geometry. Bind stored locators to `inputSha256`, source identity, text hash/view and the record ids; do not resolve them against another run by phrase alone. `parseArtifactJsonl(rawJsonl, {inputSha256, expectedSourceId})` validates manifest-first batch -ordering, schema `0.14.x`/`0.15.x`/`0.16.x`/`0.17.x`, input/provenance identity, text hashes, range boundaries, +ordering, schema `0.14.x` through `0.19.x`, input/provenance identity, text hashes, range boundaries, full source-map coverage, referenced artifact sets and page relationships. It rejects truncated/malformed streams, forged synthetic text and mismatched source views. This is a batch reader; use `extractPdfPages` for lifecycle streams. @@ -139,7 +145,7 @@ views. This is a batch reader; use `extractPdfPages` for lifecycle streams. extraction adapter always binds it to the input hash supplied to the CLI. For `0.14.x` only, chunks can carry an internal label when manifest/provenance source identity is null. The reader retains that label as metadata; it cannot -override manifest, input hash or provenance identity. Current `0.15.x`–`0.17.x` records +override manifest, input hash or provenance identity. Current `0.15.x`–`0.19.x` records must agree with the manifest, including null when no source id was supplied. Specialist records retain their legacy shared provenance labels; uniqueness is @@ -147,7 +153,7 @@ checked with their actual request/attempt/response/result ids. Some current tabl provenance block references describe pre-reconciliation indices. The adapter preserves those raw values and reports missing references as `unresolvedProvenanceLinks`, rather than using them for source-text locators. -Core span/block/chunk maps remain strict. In schema `0.15.x`–`0.17.x`, `document.tables` +Core span/block/chunk maps remain strict. In schema `0.15.x`–`0.19.x`, `document.tables` exposes table/cell `reconciled_block_ids` checked against owning source spans. Span `reconciled_block_id` and ownership provenance must agree with block maps; cells must share their table's page. Use these authoritative fields for links. @@ -188,6 +194,66 @@ the contributing invocations. The adapter rejects out-of-page or malformed scope specialist context. A locator's `sourceSpanId` resolves to the retained span and its scope; null scope explicitly withholds that attribution. +Schema `0.18.x` adds nullable `page_route.page_geometry`, OCR attempt +`raster_transform`, formula attempt `config.crop_transform`, and specialist +request `crop_geometry`. The reader validates declared geometry against the +page boxes, quarter-turn rotation, UserUnit and dimensions. Older supported +schema families retain their existing contract. A page's display uses the +effective CropBox in physical points; raster crops use MediaBox. Existing native, +Form, OCR and table bboxes remain unrotated raw PDF coordinates. One raw unit is +UserUnit physical points. Legacy PageInfo dimensions and the manifest's +`display_transform:null` are retained for compatibility. + +Raster contexts contain `pixel_to_pdf`, `rasterizer_dpi`, `pixel_width` and +`pixel_height`. The pixel matrix maps top-left crop-local pixel edges to raw PDF +coordinates. Keep it with accepted crop evidence; do not infer it from a bbox +and requested DPI. Whole-page OCR maps accepted dimensions to exact MediaBox +extents, while regional crops retain their outward pixel-grid extent. Formula +child requests receive accepted crop geometry; public crop paths and request +geometry remain redacted. Quarter-turn support follows declared page rotation +and does not detect scan orientation. Invalid or unsupported geometry remains +unknown and does not relax pixel, byte, region or attempt caps. + +On rotated OCR pages, inferred tables do not replace the accepted OCR word +sequence or become table artifacts. The raw coordinate axes do not establish +the recognizer's rows and columns; rotated table reconstruction remains pending. + +Schema `0.19.x` retains the geometry, paint and content-scope checks from earlier +versions and requires nullable `vector_evidence` on page and region routes. +Known page summaries expose bounded collection status and classified shape +counts, operator counts and cumulative lexer-input bytes. Declared bounds are +512 shapes, 512 segments per path, 100,000 operators, 8 MiB of vector-lexer +input across page and Forms, graphics depth 32, 256 Form invocations and +64 candidates. Initial PDF decompression and whole-process RSS are outside +these vector bounds. Known regional candidates expose shape/segment/subpath counts, native +coverage and `decision:"unexplained_vector"`. The reader checks the declared +caps and counters, candidate counts, reasons and same-scope OCR request links before returning +a document or delivering a streamed page. Page summaries may describe more +candidates than a retained regional subset. Explicit null evidence remains +unknown; the adapter does not infer vector text or synthesize spans. +Each regional payload also retains its exact PDF-space `bbox`, contained by +the routed bbox within serialization rounding. A larger image OCR route can +cover that candidate and reuse its request; native coverage describes the +candidate bbox rather than the larger route. +Recovery requires nonblank retained fresh OCR text with a bbox intersecting the +exact candidate extent with positive area. A completed OCR attempt that finds +text only elsewhere in a reused bitmap crop leaves that candidate unresolved. +This tests text availability; the merged bbox and extraction completeness remain +unverified. + +Vector candidates express geometric suspicion and use the existing OCR crop +path and quotas. Display axes guide grouping when page geometry is known; +source boxes and locators remain raw PDF user-space evidence. Rules, simple +boxes, native-accounted shapes and isolated logos/symbols can be withheld. +`partial` and `limited` statuses do not establish visibility or complete page +coverage. Rotated table promotion remains deferred. +Native coverage uses rectangle unions with at most 64 intersections per +shape/candidate query and 1,000,000 span checks during page classification. +Candidate-merge coverage uses its own bounded check budget. Exhaustion yields +partial evidence and withholds the affected candidate, rather than assigning +zero coverage. Captured clipping restricts native accounting; it does not prove +final pixel visibility. + ActualText replacements are preserved once per scope in `sourceText` and parser chunks. Unequal glyph/character counts retain span-bbox precision; no exact character geometry is inferred from a semantic replacement. Explicit/inferred @@ -233,11 +299,20 @@ as `memory_limit`/`output_limit`, without a complete receipt. ## Adoption and verification Readers limited to `0.11.x`/`0.13.x` must add -`0.17.x` with contract tests before changing their parser pin. The adapter retains -`0.14.x`, `0.15.x` and `0.16.x` reader support. Applications must pair +`0.19.x` with contract tests before changing their parser pin. The adapter retains +`0.14.x` through `0.18.x` reader support. Applications must pair their adopted parser revision with the selected Zig 0.17/compiler/archive identities and run its baseline CPU/runtime smoke lane. Formula children must echo the new -request schema version. Neither host is changed by this work. +request schema version. Host adoption remains a separate step. + +Schema `0.17` landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6), +merge commit `9cac608`, with all six CI checks green. The coordinate and rotated +OCR continuation is locally checked, uncommitted schema `0.18`. The `0.17` +CI result does not qualify this continuation. +Schema `0.19` adds bounded vector candidates; it is locally checked and +uncommitted. The new vector suite passes all 13 tests with no skips, including +actual OCR in both SDK formats and disjoint output in a reused bitmap crop. +Remote CI, merge and host adoption remain separate. Run the local adapter regressions and actual CLI composition test: diff --git a/docs/output-schema.md b/docs/output-schema.md index 66c149a..f57ce00 100644 --- a/docs/output-schema.md +++ b/docs/output-schema.md @@ -6,10 +6,10 @@ should depend on the records documented here. ## Version -Current schema version: `0.17.0` +Current development schema version: `0.19.0` (locally checked, uncommitted). Current parser build identity: `0.4.0-dev`. The `v0.3.0` release emitted schema -`0.11.0`; the development suffix keeps post-release schema `0.17.0` artifacts +`0.11.0`; the development suffix keeps post-release schema `0.19.0` artifacts distinguishable until the next release is tagged. The project is still pre-`1.0.0`, so incompatible schema changes may happen. @@ -34,6 +34,22 @@ JSON output is a `document_manifest` object with arrays of typed records. but records are produced page by page so host applications can persist partial artifacts and enqueue embeddings before the document finishes. +Schema `0.19.0` adds required nullable vector evidence on page and region route +traces. Bounded path collection proposes geometric OCR candidates without +changing the raw-coordinate or source-text contract. Readers must explicitly +accept `0.19.x`; formula children must echo it. This unit is locally checked and +uncommitted. Its [qualification](parser-host-implementation-plan.md#bounded-vector-routing-019-locally-checked) +is separate from earlier local results and remote CI. + +Schema `0.18.0` adds nullable page geometry, OCR raster transforms and formula +crop transforms. Native and recovered bboxes retain unrotated PDF user-space +coordinates. Quarter-turn raster mapping uses the page's declared rotation and +UserUnit; this does not detect the orientation of scanned content. Readers must +explicitly accept `0.18.x`; formula children must echo it. Schema `0.17.0` landed +in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6), merge commit +`9cac608`, with all six CI checks green. Schema `0.18.0` has local verification; +remote CI and merge remain separate. + Schema `0.17.0` adds nullable `content_scope` to spans and inline specialist span context. It identifies native page content or a particular Form invocation. Semantic replacements and inferred text boundaries survive adaptive @@ -217,6 +233,71 @@ confidence, reasons, counts, signals, bbox, and specialist metadata when available. Route traces include `specialist_request_ids` and `specialist_status` when a route maps to a request. +In `0.18`, `page_route` traces also carry nullable `page_geometry`. Known geometry +contains `coordinate_space:"pdf_user_space"`, `user_unit`, `rotation_degrees`, +`media_box`, the effective `crop_box`, and `display`. The effective CropBox is +the declared CropBox intersected with MediaBox, or MediaBox when CropBox is absent. +MediaBox, CropBox and Rotate follow page-tree inheritance; UserUnit comes from +the leaf page and defaults to 1. + +`display` contains `box:"crop_box"`, `origin:"top_left"`, `unit:"point"`, +`width`, `height`, `pdf_to_display` and `display_to_pdf`. Display dimensions are +physical points (1/72 inch), including UserUnit and the normalized quarter turn. +Matrices use PDF `[a,b,c,d,e,f]` order: `x' = a*x + c*y + e` and +`y' = b*x + d*y + f`. Map all four bbox corners and take their enclosing box. +Invalid boxes or UserUnit, an empty effective CropBox, or a non-quarter rotation +leave `page_geometry:null`; callers must withhold a display mapping in that case. + +In `0.19`, page and region route traces also carry required nullable +`vector_evidence`. Other stage traces do not carry this field. A page summary +contains `status` (`complete`, `partial` or `limited`), `shape_count`, +`unknown_count`, `ignored_rule_count`, `clipped_out_count`, +`native_accounted_count`, `simple_graphic_count`, `ambiguous_shape_count` and +`candidate_region_count`, `operator_count` and `content_bytes`. It declares +`max_shapes:512`, `max_segments_per_path:512`, `max_operators:100000`, +`max_candidate_regions:64`, `max_content_bytes:8388608`, +`max_graphics_depth:32` and `max_form_invocations:256`. Counts are nonnegative +integers; operator and content-byte counters cannot exceed their declared caps. +The content-byte budget applies cumulatively to input accepted by the vector +lexer across the page and its Forms. Initial PDF decompression is separate; +these limits do not establish a whole-process memory bound. The classified +shape counts exclude `unknown_count`, which counts unsupported or malformed +operator/path evidence independently of retained shapes. Classified shapes and +the shapes in retained candidates cannot exceed `shape_count`. + +A region candidate contains positive integer `shape_count`, `segment_count` +and `subpath_count`, exact `bbox`, `native_coverage` in `[0,1]`, and +`decision:"unexplained_vector"`. Its route is `queue_ocr`; its reasons and +provenance include `unexplained_vector`, and its specialist request ids link to +a same-page, same-region OCR request. Candidate boxes remain raw, unrotated PDF +user-space hulls. The payload `bbox` is the extent used for native coverage; +it is contained by the route bbox, allowing the route's two-decimal rounding. +A vector candidate may reuse a larger image OCR route instead of requesting a +duplicate crop. Reporting that candidate as recovered requires retained, +nonblank fresh OCR text whose bbox intersects the exact candidate bbox with +positive area. Recognized text elsewhere in the larger bitmap crop does not +satisfy that requirement. Recovery indicates text availability; the merged route +bbox and candidate completeness remain unverified. Grouping follows display axes +when known page geometry is +available. A retained collection may contain fewer candidates than the page +summary reports. + +This evidence identifies geometric candidates for the existing bounded OCR +path. It does not recognize glyphs or prove final pixel visibility or document +completeness. Rules, simple rectangular graphics, shapes accounted for by native +text, and isolated symbols/logos can be withheld. `partial` records unsupported +or uncertain evidence; `limited` records a reached work bound. Neither status +authorizes a completeness claim. `null` withholds vector attribution rather +than fabricating an empty known summary. No new OCR engine or model is implied. +Native accounting uses the union of intersecting painted span rectangles, +restricted by captured clip bounds. Each shape/candidate coverage query permits +64 intersecting rectangles, with at most 1,000,000 native-span checks during +page classification. The bounded candidate-merge coverage query uses its own +check budget. +Exhaustion makes evidence partial and withholds the affected candidate; it does +not establish zero native coverage. Rectangle unions still do not prove pixel +visibility. + ### `specialist_request` Request record for a swappable local specialist. The parser emits these records @@ -241,6 +322,10 @@ For formula execution, the public artifact remains redacted with `crop_image_path:null`; only the bounded child-process request contains the ephemeral local crop path. +Schema `0.18` requests include nullable `crop_geometry`. Formula child requests +carry this geometry after a crop is accepted; redacted public requests retain +null. The object uses the raster-context fields described below. + ### `specialist_attempt` One record per specialist invocation, including unsuccessful invocations. @@ -259,6 +344,26 @@ count. Formula attempts are selected only when the response is completed and contains one valid formula artifact. `failure_stage` distinguishes rasterize failures from recognize failures. +Schema `0.18` adds the following nullable raster contexts: + +| Record | Field | Meaning | +| --- | --- | --- | +| OCR attempt | `raster_transform` | Accepted whole-page or regional OCR image mapping | +| Formula attempt | `config.crop_transform` | Accepted formula crop mapping | +| Specialist request | `crop_geometry` | Formula crop mapping sent to the child | + +Each known context contains `pixel_to_pdf`, `rasterizer_dpi`, `pixel_width` and +`pixel_height`. The matrix maps crop-local, top-left pixel edges to unrotated +PDF user space. A missing mapping is null, not an identity transform. The +configured DPI remains the requested physical DPI; the selected Poppler backend +uses `rasterizer_dpi = requested_dpi * user_unit` because it ignores UserUnit. +Formula `crop_bbox` retains its legacy requested-region meaning. Use the pixel +matrix and dimensions to recover the actual raster extent. + +Rotated OCR words retain the recognizer's row order. Tables inferred from their +raw PDF axes are not promoted to table artifacts or replacement text until +rotated table reconstruction is qualified. + `attempt_status` is `not_invoked`, `completed`, `empty`, `unavailable`, `failed`, `timeout`, or `invalid_output`. A bounded OCR policy may emit a 200-DPI/PSM-6 attempt followed by a 300-DPI/PSM-11 attempt. Exactly one @@ -282,8 +387,9 @@ have `crop:null`; regional attempts carry `crop` with `bbox`, `dpi`, `pixel_width`, `pixel_height`, nullable `image_sha256` and `image_byte_length`. The hash and length identify the accepted PNG bytes delivered to recognition. They stay null when rasterization is refused or fails. The bbox is the actual -pixel-aligned crop in PDF user space: padded by two points, clamped to MediaBox, -then rounded outward using the nonzero page origin. Ephemeral PNG paths are +pixel-aligned crop in PDF user space: padded by two physical points, clamped to +MediaBox, then rounded outward on the raster grid. Padding in raw PDF units is +divided by UserUnit. Ephemeral PNG paths are deleted after each attempt and never exposed as public crop identity. Whole-page OCR preserves its established MediaBox coordinate mapping; crop rounding does not rescale whole-page recognition geometry. @@ -291,9 +397,11 @@ rounding does not rescale whole-page recognition geometry. Good native pages route image placements individually; pages failing native quality use whole-page recovery. Native text survives failed recovery. Quota skips retain requested routes and `not_invoked` attempts with `ocr_region_limit` -or `ocr_attempt_limit`. Pixel/byte refusal, invalid geometry and rotation have -explicit diagnostics. Rotated OCR pages are refused until their raster mapping -is qualified. Pixel bounds apply before launch and to accepted PNG dimensions; +or `ocr_attempt_limit`. Pixel/byte refusal and invalid geometry have explicit +diagnostics. Declared rotations of 0, 90, 180 and 270 degrees use shared forward +and inverse transforms. Unsupported rotations retain explicit refusal outcomes. +The rasterizer uses MediaBox for compatibility, while display geometry uses the +effective CropBox. Pixel bounds apply before launch and to accepted PNG dimensions; byte bounds apply before reading the PNG. These are not child RSS or in-flight temporary-file disk caps. @@ -373,10 +481,14 @@ IDs in batch and streaming maps are document-global. Byte ranges are local to their own exact text view, not offsets in the input PDF file or in Markdown. The manifest's `text_contract` declares the byte/range units, zero-based page -indices and geometry: `pdf_user_space`, points, `span_bbox` highlight precision, -and `display_transform:null`. Existing bbox semantics are unchanged. Neither -these boxes nor a multi-page union bbox are screen pixels. A viewer must apply -its own page rotation/crop/display transform; the contract does not supply one. +indices and compatibility geometry: `pdf_user_space`, `unit:"point"`, +`span_bbox` highlight precision, and `display_transform:null`. Existing bbox +semantics are unchanged: these boxes contain raw PDF user-space values. One raw +unit equals UserUnit physical points, so the compatibility unit label alone does +not supply physical scaling when UserUnit differs from 1. Legacy PageInfo +dimensions also remain raw. Neither these boxes nor a multi-page union bbox are +screen pixels. In `0.18`, use the matching page route's `page_geometry` to map a +box to the effective CropBox display; apply the viewer's scale separately. The origin can be native, OCR, specialist, or fallback text as its span's provenance declares; these mappings do not prove source-image visibility, complete page coverage, quotation verification, or absence of extraction loss. @@ -668,13 +780,13 @@ Minimal specialist config shape: The config format is intentionally small and bounded to 256 KiB, 32 arguments, 4 KiB per executable/argument, a 72–600 crop DPI, 0–72 points of crop padding, -and a 1–120000 ms timeout. In schema `0.13.0`, an enabled `formula` entry on a -file-backed, unrotated page receives a deterministic PNG crop. Its executable +and a 1–120000 ms timeout. An enabled `formula` entry on a file-backed page with +valid quarter-turn geometry receives a deterministic PNG crop. Its executable reads one `specialist_request` JSONL record from stdin and writes one `specialist_response` JSONL record to stdout. The parser remains model-neutral and does not depend on a specific Python package or hosted model. Memory-opened -documents and rotated pages produce explicit rasterize-stage outcomes. Other -entries remain reserved adapter configuration. +documents, unsupported rotations and invalid geometry produce explicit +rasterize-stage outcomes. Other entries remain reserved adapter configuration. ## Host Packaging Modes diff --git a/docs/parser-host-implementation-plan.md b/docs/parser-host-implementation-plan.md index e4138c7..b98cacc 100644 --- a/docs/parser-host-implementation-plan.md +++ b/docs/parser-host-implementation-plan.md @@ -1,6 +1,6 @@ # Parser, Research consumer, and Editorial consumer implementation plan -Status: phases 0/1, reconciled table links, bounded streaming and recovery assessments landed in [PR #5](https://github.com/Tom-R-Main/pdf-parser/pull/5), merge commit `0dd534d`. Paint/clipping, bounded regional OCR and the text-fidelity continuation are implemented locally in schema 0.17; verification is recorded below. Host adoption remains pending, 2026-10-10. All writes are restricted to pdf-parser: planned host edits below remain adoption instructions. Based on the [local reference review](local-reference-review-2026-10-09.md) and [host use-case review](host-usecase-reference-review-2026-10-10.md). Richer hierarchy/figures and host rollout remain planned. +Status: phases 0/1, reconciled table links, bounded streaming and recovery assessments landed in [PR #5](https://github.com/Tom-R-Main/pdf-parser/pull/5), merge commit `0dd534d`. Paint/clipping, bounded regional OCR and text fidelity landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6), merge commit `9cac608`, with all six CI checks green for schema 0.17. Coordinate transforms and rotated OCR are locally checked, uncommitted schema 0.18. Bounded vector routing is locally checked, uncommitted schema 0.19; remote CI and merge remain separate. Host adoption remains pending, 2026-10-11. All writes are restricted to pdf-parser: planned host edits below remain adoption instructions. Based on the [local reference review](local-reference-review-2026-10-09.md) and [host use-case review](host-usecase-reference-review-2026-10-10.md). Richer hierarchy/figures and host rollout remain planned. ## Delivery order and ownership @@ -142,7 +142,7 @@ Schema 0.15.0 implements three parser-side units from the later phases: Regressions exercise forged ownership, contradictory coverage, later-page source locators, actual native allocation exhaustion, mixed native/scan extraction and stream failure paths. The previous phase receipts describe historical hashes; -the continuation receipt records the current source/binary and gate results. +the continuation receipt records the source/binary and gate results for that phase. Observed checks: all 128 JavaScript tests pass without skips, strict declaration typechecking and Zig formatting pass, and native test/build targets pass all @@ -238,8 +238,9 @@ metadata lifetime defects. Both were fixed with regression tests and allocation sweeps. Review used independent contexts on the inherited session model; it does not establish model diversity. The [text-fidelity receipt](../benchmark/eval/outputs/text-fidelity-2026-10-10/receipt.json) -binds the current source and binary to the final local checks. This continuation -is uncommitted; no new remote CI, merge, host adoption or deployment is claimed. +binds the source and binary used for the final local checks. This continuation +subsequently landed in PR #6 at `9cac608`, with all six CI checks green. Host +adoption and deployment remain outside that result. Observed final local checks: 183 JavaScript tests pass without skips; native tests pass all 62 steps, shared builds 6/6 and WASM 5/5. The combined benchmark @@ -255,10 +256,113 @@ Expected injected faults remain detected. This qualifies the diagnostic native lane; Python and OCR memory lifetimes were excluded, and logical suballocation boundaries inside live arenas remain outside its proof. -Inferred next implementation unit: a shared forward/inverse transform contract -for page boxes, rotation, UserUnit and raster crops. Qualify quarter-turn pages, -nonzero origins, fractional edges and transformed Forms before enabling rotated -OCR. Then identify unexplained vector regions with new outlined-text controls and -negative cases for rules, highlights, borders and logos. Keep recovered text -separate from visual completeness. Table topology and reading order need new -held-out suites; the earlier reading-order holdout must not be retuned or promoted. +### Coordinate transforms and rotated OCR (0.18, local) + +The shared page model retains raw PDF user-space bboxes and adds a forward/inverse +display contract. Page routes carry nullable `page_geometry`: MediaBox, effective +CropBox, normalized quarter-turn rotation, leaf-page UserUnit and physical-point +display dimensions/matrices. MediaBox, CropBox and Rotate retain page-tree +inheritance. Legacy PageInfo dimensions and manifest `display_transform:null` +remain compatible. Invalid geometry or non-quarter rotations withhold the new +mapping; no identity transform is fabricated. + +OCR and formula raster callers use MediaBox. Regional crops round outward on the +backend pixel grid; whole-page OCR maps accepted bitmap dimensions to exact page +extents. The selected Poppler backend ignores UserUnit, so rasterizer DPI is the +requested physical DPI multiplied by UserUnit. Padding is divided by UserUnit +before applying it to raw coordinates. Accepted OCR attempts expose nullable +`raster_transform`; formula attempts expose nullable `config.crop_transform` and +their child requests receive `crop_geometry`. Each context retains pixel-to-PDF +matrix, effective rasterizer DPI and dimensions. Formula's legacy requested bbox +is preserved separately. + +OCR word boxes map all four corners back to raw PDF coordinates. Recognizer +word/line/block metadata retains readable boundaries through quarter turns. +This follows declared page rotation; it does not detect the orientation of scans. +Rotated OCR preserves the recognizer's word order instead of promoting tables +inferred from raw PDF axes. Rotated table reconstruction remains unqualified. +Pixel, byte, region and attempt caps remain in place. Neutral adapters expose +page geometry and explicit-scale bbox mapping without inventing character boxes. + +Local verification passed for this unit: 82 combined native/build/shared/WASM/ +benchmark steps, 241 JavaScript tests without skips, strict declarations, +36 corpus documents through both readers, and three OCR quality gates plus +their SDK readbacks. Numeric and actual raster controls cover quarter turns, +inverse mappings, negative/nonzero origins, distinct CropBox, leaf-only UserUnit, +fractional seams, transformed Forms applied once, invalid/overflow refusal, +text/chunks/locators and retained budget failures. Linux passed 19 scoped memory +checks and eight actual OCR quarter-turn CLI controls. Whole-Python-process +shutdown remains unqualified. Ten native tests were skipped; the C-enabled +recognition/teardown memory lane was exercised separately. Evidence is retained +under ignored `benchmark/eval/outputs/coordinate-transforms-2026-10-10/`. +No remote CI or merge is claimed for schema 0.18. + +### Bounded vector routing (0.19, locally checked) + +Collect painted-path hulls with limits of 512 retained shapes, 512 segments per +path and 100,000 operators. Cap cumulative vector-lexer input across page and +Forms at 8 MiB, graphics stack depth at 32 and Form invocations at 256. These +bounds exclude initial PDF decompression and do not establish whole-process +memory limits. Group unexplained contours into at most 64 regional +OCR candidates. Page summaries retain collection status, classification +counts and operator/byte counters; candidate records retain exact PDF-space +bbox, shape/segment/subpath counts and native +coverage. Required nullable `vector_evidence` appears only on page and region +route traces. Known candidates link their `unexplained_vector` reason to an +actual same-scope OCR request. Missing evidence remains null. +When an existing image OCR route covers a vector candidate, retain the exact +candidate bbox inside the larger route and reuse its request. +Require nonblank retained fresh OCR text to intersect the exact candidate bbox +with positive area before reporting recovery. Text elsewhere in the shared +bitmap crop leaves the vector candidate unresolved. Recovery is text +availability; merged route bounds and completeness remain unverified. + +Use known display axes for grouping, while public candidate boxes remain raw, +unrotated PDF coordinates. Withhold rules, simple boxes, native-accounted shapes +and isolated symbols. Unsupported evidence is partial; exhausted work bounds +are limited. Geometric suspicion does not establish glyph identity, final +visibility or completeness. Recovery uses the existing OCR engine and bounded +crop quotas. Rotated OCR table promotion remains deferred. +Native accounting unions clipped painted span rectangles, capped at 64 +intersections per shape/candidate coverage query and 1,000,000 native-span +checks during page classification. Candidate-merge coverage uses its own bounded +check budget. Exhaustion produces partial evidence and candidate abstention; +it must not establish zero coverage. + +Acceptance passed on 23 generated fixtures with real curved glyph outlines, +mixed native/vector content, transformed Forms, quarter turns, clipping, thin +rules, highlights, borders, isolated logos, ambiguous dense graphics and reached +shape/segment limits. All 47 generated files reproduce byte-for-byte; the PDFs +total 156,147 bytes. Sheared Form clipping remains an explicit abstention. + +Independent review reproduced and remediation fixed three defects: a narrow +native clip could suppress visible outlines, exhausted native accounting could +publish fabricated zero coverage, and disjoint OCR text in a reused bitmap crop +could mark its vector candidate recovered. Each has a persistent regression; +saved before/after CLI probes cover both JSONL formats. The final scoped +readback found no remaining material issue. Review used a separate context with +the inherited session model; this does not establish model diversity. + +Current local gates pass 82/82 Zig build steps and 3,774/3,784 tests, with ten +documented skips; native, shared-library and WASM builds pass. All 339 JavaScript +tests pass with no skips in a serial run, including 13 vector CLI/SDK tests. +Both SDK readers accept all 36 corpus documents and three OCR-enabled controls. +The 36-document adaptive evaluation, both born-digital/hard-font lanes and +three OCR quality gates pass. An earlier parallel JS run missed a process-test +startup deadline; the retained serial run passes that test without a product +change. No speed or universal outlined-text accuracy claim follows from these +checks. + +Linux/aarch64 passes 19 scoped parser/ABI/counter and C OCR teardown memory +checks with no skips, plus 16 vector CLI controls across both JSONL formats. +The latter exercise actual OCR children without instrumenting their memory. +Whole-Python-process shutdown remains unqualified, and `memcheck_proven` remains +false. These checks do not establish a whole-process memory bound. + +Evidence and checked binaries are retained under ignored +`benchmark/eval/outputs/vector-routing-2026-10-11/`. This unit remains +uncommitted; no remote CI or merge is claimed. + +Table topology and reading order need new held-out suites; the earlier +reading-order holdout must not be retuned or promoted. Host adoption remains a +separate change after parser contract verification. diff --git a/integrations/javascript/coordinates-integration.test.mjs b/integrations/javascript/coordinates-integration.test.mjs new file mode 100644 index 0000000..87044f7 --- /dev/null +++ b/integrations/javascript/coordinates-integration.test.mjs @@ -0,0 +1,198 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { access, readFile, mkdtemp, writeFile, rm } from 'node:fs/promises'; +import { constants } from 'node:fs'; +import { execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { extractPdfDocument, extractPdfPages, parseArtifactJsonl, locateSourceRange, + getPageGeometry, mapPdfBBoxToDisplay, mapDisplayBBoxToPdf } from './index.mjs'; + +const binaryPath = fileURLToPath(new URL('../../zig-out/bin/pdf-parser', import.meta.url)); +let available = true; +try { await access(binaryPath, constants.X_OK); } catch { available = false; } +const scan = await readFile(new URL('./fixtures/quarter-scan.jpg', import.meta.url)); +const boxes = [[10, 20, 310, 220], [10, 20, 210, 320], [10, 20, 310, 220], [10, 20, 210, 320]]; +const placements = [[300, 0, 0, 200, 10, 20], [0, 300, -200, 0, 210, 20], + [-300, 0, 0, -200, 310, 220], [0, -300, 200, 0, 10, 320]]; +// Pixels are generated text, counter-rotated so each rendered page is upright. +// This tests coordinate recovery, rather than automatic orientation detection. +function scanPdf(rotation, userUnit = 1, crop = null) { + const quarter = rotation / 90; + const content = `q ${placements[quarter].join(' ')} cm /Im1 Do Q`; + const image = Buffer.concat([Buffer.from(`<< /Type /XObject /Subtype /Image /Width 600 /Height 400 /ColorSpace /DeviceRGB /BitsPerComponent 8 /Filter /DCTDecode /Length ${scan.length} >>\nstream\n`), scan, Buffer.from('\nendstream')]); + const objects = ['<< /Type /Catalog /Pages 2 0 R >>', '<< /Type /Pages /Kids [3 0 R] /Count 1 >>', + `<< /Type /Page /Parent 2 0 R /MediaBox [${boxes[quarter].join(' ')}] /Rotate ${rotation} /UserUnit ${userUnit} ${crop ? `/CropBox [${crop.join(' ')}]` : ''} /Resources << /XObject << /Im1 5 0 R >> >> /Contents 4 0 R >>`, + `<< /Length ${Buffer.byteLength(content)} >>\nstream\n${content}\nendstream`, image]; + return pdfObjects(objects); +} +function pdfObjects(objects) { + const parts = [Buffer.from('%PDF-1.7\n')]; + const offsets = []; + let length = parts[0].length; + for (const [index, object] of objects.entries()) { + offsets.push(length); + const bytes = Buffer.concat([Buffer.from(`${index + 1} 0 obj\n`), Buffer.from(object), Buffer.from('\nendobj\n')]); + parts.push(bytes); length += bytes.length; + } + const xref = length; + parts.push(Buffer.from(`xref\n0 ${objects.length + 1}\n0000000000 65535 f \n${offsets.map(offset => `${String(offset).padStart(10, '0')} 00000 n \n`).join('')}trailer\n<< /Size ${objects.length + 1} /Root 1 0 R >>\nstartxref\n${xref}\n%%EOF\n`)); + return Buffer.concat(parts); +} +function closeBox(actual, expected, tolerance = 0.02) { + for (const [index, key] of ['x0', 'y0', 'x1', 'y1'].entries()) assert.ok(Math.abs(actual[key] - expected[index]) <= tolerance, `${key}: ${actual[key]} != ${expected[index]}`); +} +function checkText(document) { + assert.ok(document.sourceText.includes('QUARTER TURN EVIDENCE'), document.sourceText); + assert.ok(document.chunks.some(chunk => chunk.content.includes('QUARTER TURN EVIDENCE'))); + const start = document.sourceText.indexOf('QUARTER'); + const locator = locateSourceRange(document, start, start + 7, { sourceTextSha256: document.sourceTextSha256 }); + assert.ok(locator.segments.some(segment => segment.sourceSpanId && segment.bbox)); + return document.spans.find(span => span.text === 'QUARTER'); +} + +test('actual quarter-turn OCR round-trips raster words into raw PDF coordinates', { skip: !available }, async t => { + const directory = await mkdtemp(join(tmpdir(), 'pdf-parser-quarters-')); + t.after(() => rm(directory, { recursive: true, force: true })); + // Independent Poppler/Tesseract144-DPI measurements, not calculated by helpers. + const expected = [[31, 170.5, 116, 185], [45, 41, 59.5, 126], [204, 55, 289, 69.5], [160.5, 214, 175, 299]]; + for (const [quarter, rotation] of [0, 90, 180, 270].entries()) { + const bytes = scanPdf(rotation); + const path = join(directory, `quarter-${rotation}.pdf`); + await writeFile(path, bytes); + const rawJsonl = execFileSync(binaryPath, ['extract-adaptive', '--input', path, '--format', 'artifact-jsonl', '--ocr-dpi', '144', '--ocr-policy', 'single'], { encoding: 'utf8', timeout: 30000 }); + const document = parseArtifactJsonl(rawJsonl, { inputSha256: createHash('sha256').update(bytes).digest('hex') }); + const word = checkText(document); + closeBox(word.bbox, expected[quarter]); + const geometry = getPageGeometry(document, 0); + assert.equal(geometry.rotation_degrees, rotation); + const display = mapPdfBBoxToDisplay(word.bbox, geometry, { scale: 2 }); + closeBox(display, [42, 70, 212, 99]); + closeBox(mapDisplayBBoxToPdf(display, geometry, { scale: 2 }), expected[quarter]); + const attempt = document.records.find(record => record.record_type === 'specialist_attempt' && record.specialist_kind === 'ocr' && record.selected); + assert.equal(attempt.raster_transform.rasterizer_dpi, 144); + assert.equal(attempt.execution.pixel_width, 600); + assert.equal(attempt.execution.pixel_height, 400); + const pages = []; + const receipt = await extractPdfPages(bytes, { binaryPath, enableOcr: true, onPage: page => pages.push(page) }); + assert.equal(receipt.complete, true); + checkText(pages[0]); + assert.equal(getPageGeometry(pages[0], 0).rotation_degrees, rotation); + } +}); + +test('UserUnit scales physical raster and CropBox display without changing raw source boxes', { skip: !available }, async () => { + const bytes = scanPdf(90, 2, [30, 40, 190, 300]); + const document = await extractPdfDocument(bytes, { binaryPath, enableOcr: true }); + const word = checkText(document); + const geometry = getPageGeometry(document, 0); + assert.equal(geometry.user_unit, 2); + assert.equal(geometry.display.width, 520); + assert.equal(geometry.display.height, 320); + const attempt = document.records.find(record => record.record_type === 'specialist_attempt' && record.specialist_kind === 'ocr' && record.selected); + assert.equal(attempt.raster_transform.rasterizer_dpi, 400); + closeBox(word.bbox, [45, 41, 59.5, 126], 1.5); + const display = mapPdfBBoxToDisplay(word.bbox, geometry, { scale: 1.5 }); + closeBox(mapDisplayBBoxToPdf(display, geometry, { scale: 1.5 }), [word.bbox.x0, word.bbox.y0, word.bbox.x1, word.bbox.y1]); + const refused = await extractPdfDocument(bytes, { binaryPath, enableOcr: true, allowPartial: true, ocrLimits: { maxImagePixels: 1 } }); + assert.ok(refused.records.some(record => record.record_type === 'specialist_attempt' && record.errors.some(error => error.code === 'rasterizer_pixel_limit'))); + assert.equal(refused.pageOutcomes[0].completeness, 'unverified'); +}); + +test('actual formula crops share quarter-turn transforms and remove ephemeral images', { skip: !available }, async t => { + const directory = await mkdtemp(join(tmpdir(), 'pdf-parser-formula-quarters-')); + t.after(() => rm(directory, { recursive: true, force: true })); + const childPath = join(directory, 'child.mjs'); + const evidencePath = join(directory, 'child.json'); + await writeFile(childPath, `import { readFileSync, writeFileSync } from 'node:fs'; +const request = JSON.parse(readFileSync(0, 'utf8')); +const image = readFileSync(request.crop_image_path); +const transform = request.crop_geometry; +if (!transform || transform.pixel_to_pdf.length !== 6 || image.length <= 24 || image.readUInt32BE(16) !== transform.pixel_width || image.readUInt32BE(20) !== transform.pixel_height) process.exit(7); +writeFileSync(process.argv[2], JSON.stringify({ request, bytes: image.length })); +process.stdout.write(JSON.stringify({ schema_version: request.schema_version, record_type: 'specialist_response', request_id: request.request_id, specialist_id: 'coordinate-fixture', specialist_kind: 'formula', status: 'completed', formulas: [{ text: 'x^2+y^2=z^2', format: 'latex', confidence: 0.95 }] }) + '\\n'); +`); + const configPath = join(directory, 'config.json'); + await writeFile(configPath, JSON.stringify({ formula: { enabled: true, executable: process.execPath, + args: [childPath, evidencePath], crop_dpi: 144, crop_grayscale: false } })); + for (const rotation of [0, 90, 180, 270]) { + const content = 'BT /F1 12 Tf 1 0 0 1 100 240 Tm (E=mc^2++++////^^^^____) Tj 1 0 0 1 100 220 Tm (alpha+beta/gamma====) Tj 1 0 0 1 100 200 Tm (sum\\(x_i^2\\)>=delta++++) Tj ET'; + const bytes = pdfObjects(['<< /Type /Catalog /Pages 2 0 R >>', '<< /Type /Pages /Kids [3 0 R] /Count 1 >>', + `<< /Type /Page /Parent 2 0 R /MediaBox [10 20 410 320] /Rotate ${rotation} /UserUnit 2 /Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>`, + `<< /Length ${Buffer.byteLength(content)} >>\nstream\n${content}\nendstream`, '<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>']); + const path = join(directory, 'input.pdf'); + await writeFile(path, bytes); + for (const format of ['artifact-jsonl', 'stream-jsonl']) { + const raw = execFileSync(binaryPath, ['extract-adaptive', '--input', path, '--format', format, '--no-ocr', '--specialist-config', configPath], { encoding: 'utf8', timeout: 30000 }); + const records = raw.trim().split('\n').map(JSON.parse); + const attempt = records.find(record => record.record_type === 'specialist_attempt' && record.specialist_kind === 'formula' && record.attempt_status === 'completed'); + assert.ok(attempt, `rotation${rotation} ${format}`); + assert.equal(attempt.config.crop_transform.rasterizer_dpi, 288); + const evidence = JSON.parse(await readFile(evidencePath, 'utf8')); + assert.equal(evidence.request.crop_geometry.rasterizer_dpi, 288); + assert.ok(evidence.bytes > 24); + await assert.rejects(access(evidence.request.crop_image_path), { code: 'ENOENT' }); + assert.ok(records.some(record => record.record_type === 'span' && record.text === 'x^2+y^2=z^2')); + if (format === 'artifact-jsonl') parseArtifactJsonl(raw, { inputSha256: createHash('sha256').update(bytes).digest('hex') }); + } + } +}); + +test('quarter-turn OCR grid retains engine rows before derived table promotion', { skip: !available }, async t => { + const directory = await mkdtemp(join(tmpdir(), 'pdf-parser-ocr-grid-')); + t.after(() => rm(directory, { recursive: true, force: true })); + const engine = join(directory, 'engine.sh'); + const rows = ['level\tpage_num\tblock_num\tpar_num\tline_num\tword_num\tleft\ttop\twidth\theight\tconf\ttext']; + for (let row = 1; row <= 4; row++) for (let column = 1; column <= 2; column++) { + rows.push(`5\t1\t1\t1\t${row}\t${column}\t${column === 1 ? 20 : 170}\t${20 + (row - 1) * 30}\t70\t12\t99\tR${row}C${column}`); + } + await writeFile(engine, `#!/bin/sh\ncat <<'TSV'\n${rows.join('\n')}\nTSV\n`, { mode: 0o700 }); + const expected = 'R1C1 R1C2\nR2C1 R2C2\nR3C1 R3C2\nR4C1 R4C2'; + for (const rotation of [0, 90, 180, 270]) { + const bytes = scanPdf(rotation); + const path = join(directory, 'input.pdf'); + await writeFile(path, bytes); + for (const format of ['artifact-jsonl', 'stream-jsonl']) { + const raw = execFileSync(binaryPath, ['extract-adaptive', '--input', path, '--format', format, '--ocr-dpi', '144', '--ocr-policy', 'single', '--ocr-executable', engine], { encoding: 'utf8', timeout: 30000 }); + const records = raw.trim().split('\n').map(JSON.parse); + assert.equal(records.filter(record => record.record_type === 'rag_chunk').map(record => record.content).join('\n\n'), expected, `rotation${rotation} ${format}`); + if (rotation !== 0) { + assert.ok(records.filter(record => record.record_type === 'span').every(span => span.source === 'fresh_ocr')); + assert.ok(!records.some(record => record.record_type === 'table')); + } + if (format === 'artifact-jsonl') { + const document = parseArtifactJsonl(raw, { inputSha256: createHash('sha256').update(bytes).digest('hex') }); + assert.equal(document.sourceText, expected); + const start = document.sourceText.indexOf('R2C2'); + const locator = locateSourceRange(document, start, start + 4, { sourceTextSha256: document.sourceTextSha256 }); + assert.ok(locator.segments.every(segment => segment.sourceSpanId && segment.bbox)); + } + } + } +}); + +test('actual regional crop at a large origin retains native and OCR source evidence', { skip: !available }, async () => { + const origin = 10000; + const content = `BT /F1 12 Tf 1 0 0 1 ${origin + 20} ${origin + 370} Tm (Native cover text provides reliable source evidence.) Tj ET\nq 300 0 0 200 ${origin + 30} ${origin + 40} cm /Im1 Do Q`; + const image = Buffer.concat([Buffer.from(`<< /Type /XObject /Subtype /Image /Width 600 /Height 400 /ColorSpace /DeviceRGB /BitsPerComponent 8 /Filter /DCTDecode /Length ${scan.length} >>\nstream\n`), scan, Buffer.from('\nendstream')]); + const bytes = pdfObjects(['<< /Type /Catalog /Pages 2 0 R >>', '<< /Type /Pages /Kids [3 0 R] /Count 1 >>', + `<< /Type /Page /Parent 2 0 R /MediaBox [${origin} ${origin} ${origin + 400} ${origin + 400}] /CropBox [${origin + 10} ${origin + 20} ${origin + 390} ${origin + 390}] /UserUnit 2 /Resources << /XObject << /Im1 5 0 R >> /Font << /F1 6 0 R >> >> /Contents 4 0 R >>`, + `<< /Length ${Buffer.byteLength(content)} >>\nstream\n${content}\nendstream`, image, + '<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>']); + function check(document) { + const word = checkText(document); + assert.ok(document.sourceText.includes('Native cover text')); + assert.ok(word.bbox.x0 > origin && word.bbox.y0 > origin); + const attempt = document.records.find(record => record.record_type === 'specialist_attempt' && record.specialist_kind === 'ocr' && record.selected); + assert.ok(Number.isInteger(attempt.region_index)); + assert.equal(attempt.raster_transform.rasterizer_dpi, 400); + const geometry = getPageGeometry(document, 0); + const mapped = mapPdfBBoxToDisplay(word.bbox, geometry, { scale: 1.25 }); + closeBox(mapDisplayBBoxToPdf(mapped, geometry, { scale: 1.25 }), Object.values(word.bbox)); + } + check(await extractPdfDocument(bytes, { binaryPath, enableOcr: true })); + const receipt = await extractPdfPages(bytes, { binaryPath, enableOcr: true, onPage: check }); + assert.equal(receipt.complete, true); +}); diff --git a/integrations/javascript/coordinates.d.mts b/integrations/javascript/coordinates.d.mts new file mode 100644 index 0000000..30a9b59 --- /dev/null +++ b/integrations/javascript/coordinates.d.mts @@ -0,0 +1,32 @@ +import type { PdfBBox, RichPdfDocument } from './extract.mjs'; + +/** PDF affine order [a,b,c,d,e,f]: x'=a*x+c*y+e, y'=b*x+d*y+f. */ +export type AffineMatrix = readonly [number, number, number, number, number, number]; +export interface PageGeometry { + coordinate_space: 'pdf_user_space'; + user_unit: number; + rotation_degrees: 0 | 90 | 180 | 270; + media_box: PdfBBox; + crop_box: PdfBBox; + display: { + box: 'crop_box'; + origin: 'top_left'; + unit: 'point'; + width: number; + height: number; + pdf_to_display: AffineMatrix; + display_to_pdf: AffineMatrix; + }; +} +export interface RasterTransform { + pixel_to_pdf: AffineMatrix; + rasterizer_dpi: number; + pixel_width: number; + pixel_height: number; +} +/** Display points multiplied by this explicit host-selected scale. */ +export interface DisplayScale { scale: number } +export interface DisplayBBox { x0: number; y0: number; x1: number; y1: number } +export function getPageGeometry(document: RichPdfDocument, pageIndex: number): PageGeometry | null; +export function mapPdfBBoxToDisplay(bbox: PdfBBox, geometry: PageGeometry, options: DisplayScale): DisplayBBox; +export function mapDisplayBBoxToPdf(bbox: DisplayBBox, geometry: PageGeometry, options: DisplayScale): PdfBBox; diff --git a/integrations/javascript/coordinates.mjs b/integrations/javascript/coordinates.mjs new file mode 100644 index 0000000..1142e75 --- /dev/null +++ b/integrations/javascript/coordinates.mjs @@ -0,0 +1,135 @@ +const object = value => value !== null && typeof value === 'object' && !Array.isArray(value); +const finiteBox = box => object(box) && ['x0', 'y0', 'x1', 'y1'].every(key => Number.isFinite(box[key])) && + box.x0 <= box.x1 && box.y0 <= box.y1 && Number.isFinite(box.x1 - box.x0) && Number.isFinite(box.y1 - box.y0); +const close = (actual, expected) => Number.isFinite(actual) && Number.isFinite(expected) && + Math.abs(actual - expected) <= 64 * Number.EPSILON * Math.max(1, Math.abs(expected)); +const sameMatrix = (actual, expected) => Array.isArray(actual) && actual.length === 6 && actual.every((value, index) => close(value, expected[index])); + +function inverse(matrix) { + if (!Array.isArray(matrix) || matrix.length !== 6 || !matrix.every(Number.isFinite)) throw new RangeError('Invalid affine matrix'); + const [a, b, c, d, e, f] = matrix; + const determinant = a * d - b * c; + if (!Number.isFinite(determinant) || determinant === 0) throw new RangeError('Noninvertible affine matrix'); + const result = [d / determinant, -b / determinant, -c / determinant, a / determinant, + (c * f - d * e) / determinant, (b * e - a * f) / determinant]; + if (!result.every(Number.isFinite)) throw new RangeError('Invalid affine inverse'); + return result; +} + +function mapBBox(matrix, bbox) { + if (!finiteBox(bbox)) throw new RangeError('Invalid bbox'); + const [a, b, c, d, e, f] = matrix; + const points = [[bbox.x0, bbox.y0], [bbox.x0, bbox.y1], [bbox.x1, bbox.y0], [bbox.x1, bbox.y1]] + .map(([x, y]) => [a * x + c * y + e, b * x + d * y + f]); + if (points.some(point => !point.every(Number.isFinite))) throw new RangeError('Nonfinite mapped bbox'); + return { x0: Math.min(...points.map(point => point[0])), y0: Math.min(...points.map(point => point[1])), + x1: Math.max(...points.map(point => point[0])), y1: Math.max(...points.map(point => point[1])) }; +} + +/** Internal reader check. Matrices must describe the declared PDF page exactly. */ +export function validatePageGeometry(geometry) { + if (!object(geometry) || geometry.coordinate_space !== 'pdf_user_space' || + !Number.isFinite(geometry.user_unit) || geometry.user_unit <= 0 || geometry.user_unit > 75000 || + ![0, 90, 180, 270].includes(geometry.rotation_degrees) || !finiteBox(geometry.media_box) || !finiteBox(geometry.crop_box)) throw new RangeError('Invalid page geometry'); + const media = geometry.media_box; + const crop = geometry.crop_box; + if (media.x0 >= media.x1 || media.y0 >= media.y1 || crop.x0 >= crop.x1 || crop.y0 >= crop.y1 || + crop.x0 < media.x0 || crop.y0 < media.y0 || crop.x1 > media.x1 || crop.y1 > media.y1) throw new RangeError('Invalid page boxes'); + const u = geometry.user_unit; + const matrices = { + 0: [u, 0, 0, -u, -u * crop.x0, u * crop.y1], + 90: [0, u, u, 0, -u * crop.y0, -u * crop.x0], + 180: [-u, 0, 0, u, u * crop.x1, -u * crop.y0], + 270: [0, -u, -u, 0, u * crop.y1, u * crop.x1], + }; + const matrix = matrices[geometry.rotation_degrees]; + const sideways = [90, 270].includes(geometry.rotation_degrees); + const width = (sideways ? crop.y1 - crop.y0 : crop.x1 - crop.x0) * u; + const height = (sideways ? crop.x1 - crop.x0 : crop.y1 - crop.y0) * u; + const display = geometry.display; + if (!(width > 0 && height > 0) || !object(display) || display.box !== 'crop_box' || display.origin !== 'top_left' || display.unit !== 'point' || + !close(display.width, width) || !close(display.height, height) || !sameMatrix(display.pdf_to_display, matrix) || + !sameMatrix(display.display_to_pdf, inverse(matrix))) throw new RangeError('Inconsistent page display transform'); + return geometry; +} + +/** Internal reader check; returns the PDF bbox of the accepted pixel image. */ +export function validateRasterTransform(transform) { + if (!object(transform) || !Number.isSafeInteger(transform.pixel_width) || !Number.isSafeInteger(transform.pixel_height) || + transform.pixel_width < 1 || transform.pixel_height < 1 || transform.pixel_width > 0x7fffffff || transform.pixel_height > 0x7fffffff || + !Number.isFinite(transform.rasterizer_dpi) || transform.rasterizer_dpi <= 0) throw new RangeError('Invalid raster transform'); + inverse(transform.pixel_to_pdf); + const bbox = mapBBox(transform.pixel_to_pdf, { x0: 0, y0: 0, x1: transform.pixel_width, y1: transform.pixel_height }); + if (bbox.x0 >= bbox.x1 || bbox.y0 >= bbox.y1) throw new RangeError('Empty raster transform'); + return bbox; +} + +/** Internal reader check when page and requested raster settings are known. */ +export function validateRasterPageContext(transform, geometry, requestedDpi, { fullPage = false } = {}) { + validatePageGeometry(geometry); + const mapped = validateRasterTransform(transform); + if (!Number.isFinite(requestedDpi) || requestedDpi <= 0 || !close(transform.rasterizer_dpi, requestedDpi * geometry.user_unit)) throw new RangeError('Inconsistent raster DPI'); + const sideways = [90, 270].includes(geometry.rotation_degrees); + const media = geometry.media_box; + if (fullPage) { + // Match the native/Poppler extent arithmetic before rounding. Dividing DPI + // first can turn a letter page's 3300-pixel height into 3300 + epsilon. + const width = Math.ceil((sideways ? media.y1 - media.y0 : media.x1 - media.x0) * transform.rasterizer_dpi / 72); + const height = Math.ceil((sideways ? media.x1 - media.x0 : media.y1 - media.y0) * transform.rasterizer_dpi / 72); + if (transform.pixel_width !== width || transform.pixel_height !== height) throw new RangeError('Inconsistent full-page raster dimensions'); + } else { + if (mapped.x1 <= media.x0 || mapped.y1 <= media.y0 || mapped.x0 >= media.x1 || mapped.y0 >= media.y1) throw new RangeError('Regional raster does not intersect the MediaBox'); + const pixelX = Math.abs(transform.pixel_to_pdf[0]) + Math.abs(transform.pixel_to_pdf[2]); + const pixelY = Math.abs(transform.pixel_to_pdf[1]) + Math.abs(transform.pixel_to_pdf[3]); + for (const key of ['x0', 'y0', 'x1', 'y1']) { + const outside = key[1] === '0' ? media[key] - mapped[key] : mapped[key] - media[key]; + const tolerance = 64 * Number.EPSILON * Math.max(1, Math.abs(media[key]), Math.abs(mapped[key])); + if (outside >= (key[0] === 'x' ? pixelX : pixelY) + tolerance) throw new RangeError('Regional raster exceeds the MediaBox'); + } + } + const bbox = fullPage ? media : mapped; + // Regional coefficients come from the DPI grid. Recovering them by + // subtracting large translated corners loses precision at page origins. + const sx = fullPage ? (sideways ? bbox.y1 - bbox.y0 : bbox.x1 - bbox.x0) / transform.pixel_width : 72 / transform.rasterizer_dpi; + const sy = fullPage ? (sideways ? bbox.x1 - bbox.x0 : bbox.y1 - bbox.y0) / transform.pixel_height : 72 / transform.rasterizer_dpi; + const matrices = { + 0: [sx, 0, 0, -sy, bbox.x0, bbox.y1], + 90: [0, sx, sy, 0, bbox.x0, bbox.y0], + 180: [-sx, 0, 0, sy, bbox.x1, bbox.y0], + 270: [0, -sx, -sy, 0, bbox.x1, bbox.y1], + }; + if (!sameMatrix(transform.pixel_to_pdf, matrices[geometry.rotation_degrees])) throw new RangeError('Inconsistent raster page orientation'); +} + +/** Returns explicit geometry when available, otherwise null for legacy/null records. */ +export function getPageGeometry(document, pageIndex) { + if (!Number.isSafeInteger(pageIndex) || pageIndex < 0 || !object(document) || !Array.isArray(document.records) || + !Number.isSafeInteger(document.manifest?.page_count) || pageIndex >= document.manifest.page_count) throw new RangeError('Invalid page index'); + const routes = document.records.filter(record => record.record_type === 'route_trace' && record.trace_kind === 'page_route' && record.page_index === pageIndex); + if (routes.length > 1) throw new RangeError('Ambiguous page geometry'); + const geometry = routes[0]?.page_geometry; + return geometry == null ? null : validatePageGeometry(geometry); +} + +function displayScale(options) { + const scale = options?.scale; + if (!Number.isFinite(scale) || scale <= 0) throw new RangeError('An explicit positive display scale is required'); + return scale; +} + +/** Display points times caller-selected scale; does not mutate PDF-space evidence. */ +export function mapPdfBBoxToDisplay(bbox, geometry, options) { + validatePageGeometry(geometry); + const scale = displayScale(options); + const mapped = mapBBox(geometry.display.pdf_to_display, bbox); + const result = Object.fromEntries(Object.entries(mapped).map(([key, value]) => [key, value * scale])); + if (!finiteBox(result)) throw new RangeError('Nonfinite display bbox'); + return result; +} + +export function mapDisplayBBoxToPdf(bbox, geometry, options) { + validatePageGeometry(geometry); + const scale = displayScale(options); + if (!finiteBox(bbox)) throw new RangeError('Invalid display bbox'); + return mapBBox(geometry.display.display_to_pdf, Object.fromEntries(Object.entries(bbox).map(([key, value]) => [key, value / scale]))); +} diff --git a/integrations/javascript/coordinates.test.mjs b/integrations/javascript/coordinates.test.mjs new file mode 100644 index 0000000..5ec970b --- /dev/null +++ b/integrations/javascript/coordinates.test.mjs @@ -0,0 +1,140 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { getPageGeometry, mapPdfBBoxToDisplay, mapDisplayBBoxToPdf } from './index.mjs'; +import { validateRasterPageContext, validateRasterTransform } from './coordinates.mjs'; + +const crop = { x0: 10, y0: 20, x1: 110, y1: 220 }; +const pdf = { x0: 20, y0: 40, x1: 50, y1: 80 }; +const cases = [ + [0, [2, 0, 0, -2, -20, 440], [0.5, 0, 0, -0.5, 10, 220], 200, 400, { x0: 30, y0: 420, x1: 120, y1: 540 }], + [90, [0, 2, 2, 0, -40, -20], [0, 0.5, 0.5, 0, 10, 20], 400, 200, { x0: 60, y0: 30, x1: 180, y1: 120 }], + [180, [-2, 0, 0, 2, 220, -40], [-0.5, 0, 0, 0.5, 110, 20], 200, 400, { x0: 180, y0: 60, x1: 270, y1: 180 }], + [270, [0, -2, -2, 0, 440, 220], [0, -0.5, -0.5, 0, 110, 220], 400, 200, { x0: 420, y0: 180, x1: 540, y1: 270 }], +]; +function geometry([rotation, forward, inverse, width, height]) { + return { coordinate_space: 'pdf_user_space', user_unit: 2, rotation_degrees: rotation, + media_box: { x0: -10, y0: -20, x1: 200, y1: 300 }, crop_box: { ...crop }, + display: { box: 'crop_box', origin: 'top_left', unit: 'point', width, height, pdf_to_display: forward, display_to_pdf: inverse } }; +} +for (const values of cases) test(`literal ${values[0]} degree display bbox with CropBox UserUnit and explicit scale`, () => { + const page = geometry(values); + const original = structuredClone({ page, pdf }); + const mapped = mapPdfBBoxToDisplay(pdf, page, { scale: 1.5 }); + assert.deepEqual(mapped, values[5]); + assert.deepEqual(mapDisplayBBoxToPdf(mapped, page, { scale: 1.5 }), pdf); + assert.deepEqual({ page, pdf }, original); +}); + +test('page geometry lookup preserves null legacy metadata and page identity', () => { + const page = geometry(cases[0]); + const document = { manifest: { page_count: 2 }, records: [{ record_type: 'route_trace', trace_kind: 'page_route', page_index: 1, page_geometry: page }] }; + assert.equal(getPageGeometry(document, 0), null); + assert.equal(getPageGeometry(document, 1), page); + document.records[0].page_geometry = null; + assert.equal(getPageGeometry(document, 1), null); + assert.throws(() => getPageGeometry(document, 2), RangeError); + assert.throws(() => getPageGeometry(document, 0.5), RangeError); + document.records.push({ ...document.records[0] }); + assert.throws(() => getPageGeometry(document, 1), /Ambiguous/u); +}); + +test('coordinate helpers refuse implicit scales invalid boxes and forged matrices', () => { + const page = geometry(cases[0]); + for (const scale of [undefined, 0, -1, Infinity, NaN, '1']) { + assert.throws(() => mapPdfBBoxToDisplay(pdf, page, { scale }), RangeError); + assert.throws(() => mapDisplayBBoxToPdf(pdf, page, { scale }), RangeError); + } + assert.throws(() => mapPdfBBoxToDisplay({ ...pdf, x0: NaN }, page, { scale: 1 }), RangeError); + assert.throws(() => mapDisplayBBoxToPdf({ ...pdf, x1: 0 }, page, { scale: 1 }), RangeError); + const forged = structuredClone(page); + forged.display.pdf_to_display[4] += 1; + assert.throws(() => mapPdfBBoxToDisplay(pdf, forged, { scale: 1 }), RangeError); + assert.throws(() => mapPdfBBoxToDisplay(pdf, null, { scale: 1 }), RangeError); +}); + +test('regional raster coefficients stay precise at large positive and negative page origins', () => { + for (const origin of [10000, -10000, 1000000]) for (const unit of [1, 2]) { + const box = { x0: origin, y0: -origin, x1: origin + 100, y1: -origin + 200 }; + const u = unit; + const v = 1 / u; + const pixel = 72 / (200 * u); + const rotations = [ + [0, [u, 0, 0, -u, -u * box.x0, u * box.y1], [v, 0, 0, -v, box.x0, box.y1], [pixel, 0, 0, -pixel, box.x0, box.y1]], + [90, [0, u, u, 0, -u * box.y0, -u * box.x0], [0, v, v, 0, box.x0, box.y0], [0, pixel, pixel, 0, box.x0, box.y0]], + [180, [-u, 0, 0, u, u * box.x1, -u * box.y0], [-v, 0, 0, v, box.x1, box.y0], [-pixel, 0, 0, pixel, box.x1, box.y0]], + [270, [0, -u, -u, 0, u * box.y1, u * box.x1], [0, -v, -v, 0, box.x1, box.y1], [0, -pixel, -pixel, 0, box.x1, box.y1]], + ]; + for (const [rotation, forward, inverse, matrix] of rotations) { + const sideways = [90, 270].includes(rotation); + const page = { coordinate_space: 'pdf_user_space', user_unit: unit, rotation_degrees: rotation, + media_box: box, crop_box: box, display: { box: 'crop_box', origin: 'top_left', unit: 'point', + width: (sideways ? 200 : 100) * unit, height: (sideways ? 100 : 200) * unit, + pdf_to_display: forward, display_to_pdf: inverse } }; + const raster = { pixel_to_pdf: matrix, rasterizer_dpi: 200 * unit, pixel_width: 17, pixel_height: 19 }; + assert.doesNotThrow(() => validateRasterTransform(raster)); + assert.doesNotThrow(() => validateRasterPageContext(raster, page, 200)); + } + } +}); + +function pageAt(box, rotation = 0, unit = 1) { + const u = unit; + const v = 1 / u; + const matrices = { + 0: [[u, 0, 0, -u, -u * box.x0, u * box.y1], [v, 0, 0, -v, box.x0, box.y1]], + 90: [[0, u, u, 0, -u * box.y0, -u * box.x0], [0, v, v, 0, box.x0, box.y0]], + 180: [[-u, 0, 0, u, u * box.x1, -u * box.y0], [-v, 0, 0, v, box.x1, box.y0]], + 270: [[0, -u, -u, 0, u * box.y1, u * box.x1], [0, -v, -v, 0, box.x1, box.y1]], + }; + const sideways = [90, 270].includes(rotation); + return { coordinate_space: 'pdf_user_space', user_unit: unit, rotation_degrees: rotation, media_box: box, crop_box: box, + display: { box: 'crop_box', origin: 'top_left', unit: 'point', width: (sideways ? box.y1 - box.y0 : box.x1 - box.x0) * unit, + height: (sideways ? box.x1 - box.x0 : box.y1 - box.y0) * unit, pdf_to_display: matrices[rotation][0], display_to_pdf: matrices[rotation][1] } }; +} + +test('regional rasters reject off-page translated extents beyond pixel rounding', () => { + const page = pageAt({ x0: 10000, y0: -10000, x1: 10100, y1: -9800 }); + const base = { pixel_to_pdf: [0.36, 0, 0, -0.36, 10000, -9800], rasterizer_dpi: 200, pixel_width: 17, pixel_height: 19 }; + for (const matrix of [ + [0.36, 0, 0, -0.36, 10200, -9800], + [0.36, 0, 0, -0.36, 9999.28, -9800], + [0.36, 0, 0, -0.36, 10000, -9799.28], + [0.36, 0, 0, -0.36, 10094.60, -9800], + [0.36, 0, 0, -0.36, 10000, -9993.88], + ]) { + const raster = { ...base, pixel_to_pdf: matrix }; + assert.doesNotThrow(() => validateRasterTransform(raster)); + assert.throws(() => validateRasterPageContext(raster, page, 200), RangeError); + } + const outsidePixel = { ...base, pixel_width: 1, pixel_to_pdf: [0.36, 0, 0, -0.36, 10100, -9800] }; + assert.throws(() => validateRasterPageContext(outsidePixel, page, 200), RangeError); +}); + +test('regional raster MediaBox edges allow fractional outward pixels with large-origin cancellation', () => { + const page = pageAt({ x0: 1000000, y0: -1000000, x1: 1000100, y1: -999800 }); + const raster = { pixel_to_pdf: [0.36, 0, 0, -0.36, 1000094.04, -999999.6], rasterizer_dpi: 200, pixel_width: 17, pixel_height: 2 }; + // The right and bottom edges extend by .16 and .32 PDF units, each < .36. + assert.doesNotThrow(() => validateRasterPageContext(raster, page, 200)); +}); + +test('full-page rasters reject dimension forgery even when their matrix maps the MediaBox', () => { + const page = pageAt({ x0: 0, y0: 0, x1: 612, y1: 792 }); + for (const [width, height] of [[2549, 3300], [2550, 3301], [1, 1]]) { + const raster = { pixel_to_pdf: [612 / width, 0, 0, -792 / height, 0, 792], rasterizer_dpi: 300, pixel_width: width, pixel_height: height }; + assert.throws(() => validateRasterPageContext(raster, page, 300, { fullPage: true }), RangeError); + } +}); + +test('300 DPI letter and fractional full pages retain exact accepted dimensions and PDF bounds', () => { + for (const rotation of [0, 90, 180, 270]) { + const page = pageAt({ x0: 0, y0: 0, x1: 612, y1: 792 }, rotation); + const sideways = [90, 270].includes(rotation); + const matrices = { 0: [0.24, 0, 0, -0.24, 0, 792], 90: [0, 0.24, 0.24, 0, 0, 0], + 180: [-0.24, 0, 0, 0.24, 612, 0], 270: [0, -0.24, -0.24, 0, 612, 792] }; + const raster = { pixel_to_pdf: matrices[rotation], rasterizer_dpi: 300, pixel_width: sideways ? 3300 : 2550, pixel_height: sideways ? 2550 : 3300 }; + assert.doesNotThrow(() => validateRasterPageContext(raster, page, 300, { fullPage: true })); + } + const box = { x0: 10.25, y0: -20.5, x1: 20.35, y1: -0.25 }; + const fractional = { pixel_to_pdf: [(20.35 - 10.25) / 43, 0, 0, -20.25 / 85, 10.25, -0.25], rasterizer_dpi: 300, pixel_width: 43, pixel_height: 85 }; + assert.doesNotThrow(() => validateRasterPageContext(fractional, pageAt(box), 300, { fullPage: true })); +}); diff --git a/integrations/javascript/extract.d.mts b/integrations/javascript/extract.d.mts index 137ac94..73ca9ea 100644 --- a/integrations/javascript/extract.d.mts +++ b/integrations/javascript/extract.d.mts @@ -1,3 +1,5 @@ +import type { PageGeometry, RasterTransform } from './coordinates.mjs'; + export type ExtractionErrorCode = 'protocol' | 'unavailable' | 'timeout' | 'output_limit' | 'memory_limit' | 'aborted' | 'crashed' | 'document' | 'cleanup' | 'callback'; export class PdfExtractionError extends Error { readonly code: ExtractionErrorCode; @@ -86,6 +88,76 @@ export interface PaintEvidence { clip_state: 'unclipped' | 'inside' | 'outside' | 'partial' | 'unknown'; clip_bbox: PdfBBox | null; } +export interface PageRouteRecord extends ArtifactRecord { + record_type: 'route_trace'; + trace_kind: 'page_route'; + page_index: number; + /** Available from 0.18; null when geometry is unavailable. */ + page_geometry?: PageGeometry | null; + /** Available from 0.19; bounded path evidence, not recognized text or visibility. */ + vector_evidence?: PageVectorEvidence | null; +} +export interface PageVectorEvidence { + status: 'complete' | 'partial' | 'limited'; + shape_count: number; + /** Unknown operators/paths; independent of the retained shape count. */ + unknown_count: number; + ignored_rule_count: number; + clipped_out_count: number; + native_accounted_count: number; + simple_graphic_count: number; + ambiguous_shape_count: number; + candidate_region_count: number; + max_shapes: 512; + max_segments_per_path: 512; + max_operators: 100000; + operator_count: number; + content_bytes: number; + max_content_bytes: 8388608; + max_graphics_depth: 32; + max_form_invocations: 256; + max_candidate_regions: 64; +} +export interface RegionVectorEvidence { + /** Exact candidate extent; may be smaller than a reused image OCR route. */ + bbox: PdfBBox; + shape_count: number; + segment_count: number; + subpath_count: number; + native_coverage: number; + decision: 'unexplained_vector'; +} +export interface RegionRouteRecord extends ArtifactRecord { + record_type: 'route_trace'; + trace_kind: 'region_route'; + page_index: number; + region_index: number; + /** Available from 0.19; null for routes without a vector candidate. */ + vector_evidence?: RegionVectorEvidence | null; +} +export interface OcrAttemptRecord extends ArtifactRecord { + record_type: 'specialist_attempt'; + specialist_kind: 'ocr'; + page_index: number; + /** Available from 0.18; null before a raster is accepted. */ + raster_transform?: RasterTransform | null; +} +export interface FormulaAttemptRecord extends ArtifactRecord { + record_type: 'specialist_attempt'; + specialist_kind: 'formula'; + page_index: number; + config: { + /** Available from 0.18; requested PDF bbox remains separate. */ + crop_transform?: RasterTransform | null; + [key: string]: unknown; + }; +} +export interface SpecialistRequestRecord extends ArtifactRecord { + record_type: 'specialist_request'; + page_index: number; + /** Available from 0.18; null unless the request has an accepted crop. */ + crop_geometry?: RasterTransform | null; +} export interface ByteSourceSegment { text_start_byte: number; text_end_byte: number; diff --git a/integrations/javascript/extract.mjs b/integrations/javascript/extract.mjs index db79a04..552c077 100644 --- a/integrations/javascript/extract.mjs +++ b/integrations/javascript/extract.mjs @@ -5,6 +5,7 @@ import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { terminateSubprocess } from './subprocess.mjs'; import { ocrArguments } from './ocr-options.mjs'; +import { validatePageGeometry, validateRasterTransform, validateRasterPageContext } from './coordinates.mjs'; export class PdfExtractionError extends Error { constructor(code, message, options) { @@ -94,7 +95,7 @@ export function parseArtifactJsonl(rawJsonl, { inputSha256, expectedSourceId } = }); const manifest = records[0]; if (!object(manifest) || manifest.record_type !== 'document_manifest') fail('Expected manifest first'); - if (!/^0\.(14|15|16|17)\.\d+$/.test(manifest.schema_version ?? '') || manifest.schema_name !== 'document_manifest' || + if (!/^0\.(14|15|16|17|18|19)\.\d+$/.test(manifest.schema_version ?? '') || manifest.schema_name !== 'document_manifest' || typeof manifest.document_id !== 'string' || !manifest.document_id || !integer(manifest.page_count) || manifest.input_sha256 !== inputSha256) fail('Unsupported schema or manifest input identity'); if (!(manifest.source_id === null || (typeof manifest.source_id === 'string' && manifest.source_id.length > 0)) || @@ -221,7 +222,55 @@ export function parseArtifactJsonl(rawJsonl, { inputSha256, expectedSourceId } = } const pageOutcomes = records.filter(record => record.record_type === 'route_trace' && record.trace_kind === 'page_route') .map(record => ({ pageIndex: record.page_index, ...record.coverage })); - if (/^0\.(16|17)\./u.test(manifest.schema_version)) { + if (/^0\.(18|19)\./u.test(manifest.schema_version)) { + const pageGeometry = new Map(); + const boxClose = (actual, expected, marginX = 0, marginY = 0) => object(expected) && + ['x0', 'y0', 'x1', 'y1'].every(key => Number.isFinite(expected[key]) && + Math.abs(actual[key] - expected[key]) <= (key[0] === 'x' ? marginX : marginY) + 0.0055 + 64 * Number.EPSILON * Math.max(1, Math.abs(expected[key]))); + const check = callback => { try { return callback(); } catch { fail('Invalid coordinate transform contract'); } }; + for (const record of records.filter(record => record.record_type === 'route_trace' && record.trace_kind === 'page_route')) { + if (record.page_geometry === undefined) fail('Missing page geometry contract'); + if (record.page_geometry !== null) pageGeometry.set(record.page_index, check(() => validatePageGeometry(record.page_geometry))); + } + for (const record of records) { + const geometry = pageGeometry.get(record.page_index); + if (record.record_type === 'specialist_attempt' && record.specialist_kind === 'ocr') { + const transform = record.raster_transform; + const execution = record.execution; + if (transform === undefined || !object(execution) || !integer(execution.pixel_width) || !integer(execution.pixel_height)) fail('Missing OCR raster transform contract'); + if (transform === null) { + if (['completed', 'empty'].includes(record.attempt_status) || execution.pixel_width !== 0 || execution.pixel_height !== 0) fail('Missing accepted OCR raster transform'); + continue; + } + const mapped = check(() => validateRasterTransform(transform)); + if (transform.pixel_width !== execution.pixel_width || transform.pixel_height !== execution.pixel_height || + (record.crop !== null && (!object(record.crop) || transform.pixel_width !== record.crop.pixel_width || transform.pixel_height !== record.crop.pixel_height)) || + !boxClose(mapped, record.crop?.bbox ?? record.provenance.bbox)) fail('Inconsistent OCR raster evidence'); + if (geometry) check(() => validateRasterPageContext(transform, geometry, record.config?.dpi, { fullPage: record.region_index === null })); + } else if (record.record_type === 'specialist_attempt' && record.specialist_kind === 'formula') { + if (!object(record.config) || typeof record.config.crop_backed !== 'boolean') fail('Invalid formula crop config'); + const transform = record.config.crop_transform; + if (transform === undefined) fail('Missing formula crop transform contract'); + if (transform === null) { + if (record.config.crop_backed === true) fail('Missing accepted formula crop transform'); + continue; + } + const mapped = check(() => validateRasterTransform(transform)); + const box = record.config.crop_bbox; + const matrix = transform.pixel_to_pdf; + const marginX = Math.abs(matrix[0]) + Math.abs(matrix[2]); + const marginY = Math.abs(matrix[1]) + Math.abs(matrix[3]); + if (record.config.crop_backed !== true || transform.pixel_width !== record.config.crop_pixel_width || transform.pixel_height !== record.config.crop_pixel_height || + !boxClose(mapped, box, marginX, marginY) || mapped.x0 > box.x0 + 0.0055 || mapped.y0 > box.y0 + 0.0055 || + mapped.x1 < box.x1 - 0.0055 || mapped.y1 < box.y1 - 0.0055) fail('Inconsistent formula raster evidence'); + if (geometry) check(() => validateRasterPageContext(transform, geometry, record.config.crop_dpi)); + } else if (record.record_type === 'specialist_request') { + if (record.crop_geometry === undefined) fail('Missing request crop geometry contract'); + if (record.crop_geometry !== null) check(() => validateRasterTransform(record.crop_geometry)); + } + } + } + if (/^0\.(16|17|18|19)\./u.test(manifest.schema_version)) { const validBox = box => object(box) && ['x0', 'y0', 'x1', 'y1'].every(key => Number.isFinite(box[key])) && box.x0 <= box.x1 && box.y0 <= box.y1; for (const span of spans) { const paint = span.paint; @@ -232,7 +281,7 @@ export function parseArtifactJsonl(rawJsonl, { inputSha256, expectedSourceId } = (['inside', 'outside', 'partial'].includes(paint.clip_state) && paint.clip_bbox === null) || (paint.paint_state === 'painting' && (paint.rendering_mode === null || [3, 7].includes(paint.rendering_mode))) || (paint.paint_state === 'non_painting' && ![3, 7].includes(paint.rendering_mode))) fail('Invalid paint evidence'); - if (manifest.schema_version.startsWith('0.17.')) { + if (/^0\.(17|18|19)\./u.test(manifest.schema_version)) { const scope = span.content_scope; if (!(scope === null || (object(scope) && scope.page_index === span.page_index && integer(scope.stream_generation) && scope.stream_generation <= 65535 && integer(scope.invocation_index) && scope.invocation_index <= 0xffffffff && @@ -259,7 +308,7 @@ export function parseArtifactJsonl(rawJsonl, { inputSha256, expectedSourceId } = if (items === 'spans' && (!object(item.paint) || ['rendering_mode', 'paint_state', 'clip_state'].some(key => item.paint[key] !== original.paint[key]) || (original.paint.clip_bbox === null ? item.paint.clip_bbox !== null : !validBox(item.paint.clip_bbox) || ['x0', 'y0', 'x1', 'y1'].some(key => item.paint.clip_bbox[key] !== original.paint.clip_bbox[key])))) fail('Inconsistent specialist paint evidence'); - if (items === 'spans' && manifest.schema_version.startsWith('0.17.') && + if (items === 'spans' && /^0\.(17|18|19)\./u.test(manifest.schema_version) && (original.content_scope === null ? item.content_scope !== null : !object(item.content_scope) || ['page_index', 'stream_object_num', 'stream_generation', 'invocation_index'].some(key => item.content_scope[key] !== original.content_scope[key]))) fail('Inconsistent specialist content scope'); seen.add(item[idField]); @@ -306,7 +355,7 @@ export function parseArtifactJsonl(rawJsonl, { inputSha256, expectedSourceId } = const pageRoutes = new Set(); for (const record of records.filter(record => record.record_type === 'route_trace' && ['page_route', 'region_route'].includes(record.trace_kind))) { const assessment = record.coverage; - if (/^0\.(16|17)\./u.test(manifest.schema_version) && (!object(assessment) || !['non_painting_span_count', 'clipped_out_span_count', 'uncertain_paint_span_count'].every(key => integer(assessment[key])))) fail('Invalid paint coverage counts'); + if (/^0\.(16|17|18|19)\./u.test(manifest.schema_version) && (!object(assessment) || !['non_painting_span_count', 'clipped_out_span_count', 'uncertain_paint_span_count'].every(key => integer(assessment[key])))) fail('Invalid paint coverage counts'); if (!integer(record.page_index) || record.page_index >= manifest.page_count || !object(assessment) || !statuses.has(assessment.status) || assessment.completeness !== 'unverified' || !recoveryStates.has(assessment.recovery_state) || typeof assessment.requested_recovery !== 'boolean' || @@ -323,6 +372,57 @@ export function parseArtifactJsonl(rawJsonl, { inputSha256, expectedSourceId } = (assessment.status === 'recovered' && (assessment.recovery_state !== 'completed' || assessment.recognized_span_count === 0 || assessment.unresolved_region_count !== 0))) fail('Inconsistent recovery assessment'); } } + if (manifest.schema_version.startsWith('0.19.')) { + const classified = ['ignored_rule_count', 'clipped_out_count', 'native_accounted_count', 'simple_graphic_count', 'ambiguous_shape_count']; + const limits = { max_shapes: 512, max_segments_per_path: 512, max_operators: 100000, max_candidate_regions: 64, + max_content_bytes: 8388608, max_graphics_depth: 32, max_form_invocations: 256 }; + const validBox = box => object(box) && ['x0', 'y0', 'x1', 'y1'].every(key => Number.isFinite(box[key])) && box.x0 < box.x1 && box.y0 < box.y1; + const containedBox = (inner, outer) => validBox(inner) && validBox(outer) && ['x0', 'y0', 'x1', 'y1'].every(key => { + // Route bboxes use two decimals; candidate evidence retains exact values. + const tolerance = 0.0055 + 64 * Number.EPSILON * Math.max(1, Math.abs(inner[key]), Math.abs(outer[key])); + return key.endsWith('0') ? inner[key] >= outer[key] - tolerance : inner[key] <= outer[key] + tolerance; + }); + const pageEvidence = new Map(); + const requests = new Map(records.filter(record => record.record_type === 'specialist_request').map(record => [record.request_id, record])); + const hasReason = record => Array.isArray(record.reasons) && record.reasons.includes('unexplained_vector') && + Array.isArray(record.provenance.route_reasons) && record.provenance.route_reasons.includes('unexplained_vector'); + const routeRecords = records.filter(record => record.record_type === 'route_trace' && ['page_route', 'region_route'].includes(record.trace_kind)); + for (const record of routeRecords) { + if (record.vector_evidence === undefined) fail('Missing vector evidence'); + if (record.trace_kind !== 'page_route') continue; + const value = record.vector_evidence; + if (value !== null) { + if (!object(value) || !['complete', 'partial', 'limited'].includes(value.status) || + !['shape_count', 'unknown_count', 'candidate_region_count', 'operator_count', 'content_bytes', ...classified].every(key => integer(value[key])) || + Object.entries(limits).some(([key, expected]) => value[key] !== expected) || + value.shape_count > value.max_shapes || value.candidate_region_count > value.max_candidate_regions || + value.operator_count > value.max_operators || value.content_bytes > value.max_content_bytes || + classified.reduce((sum, key) => sum + value[key], 0) > value.shape_count || + value.candidate_region_count > value.shape_count || (value.candidate_region_count > 0 && !hasReason(record))) fail('Invalid page vector evidence'); + } + pageEvidence.set(record.page_index, { value, regions: new Set(), shapes: 0 }); + } + for (const record of routeRecords.filter(record => record.trace_kind === 'region_route' && record.vector_evidence !== null)) { + const value = record.vector_evidence; + const page = pageEvidence.get(record.page_index); + if (!object(value) || !page?.value || !integer(record.region_index) || page.regions.has(record.region_index) || + !['shape_count', 'segment_count', 'subpath_count'].every(key => integer(value[key]) && value[key] > 0) || + value.decision !== 'unexplained_vector' || !Number.isFinite(value.native_coverage) || value.native_coverage < 0 || value.native_coverage > 1 || + value.shape_count > page.value.shape_count || value.segment_count > value.shape_count * page.value.max_segments_per_path || + value.subpath_count > value.segment_count || !containedBox(value.bbox, record.bbox) || + record.route !== 'queue_ocr' || !hasReason(record) || + record.specialist_status !== 'requested' || !Array.isArray(record.specialist_request_ids) || + !record.specialist_request_ids.some(id => { + const request = requests.get(id); + return request?.requested_kind === 'ocr' && request.page_index === record.page_index && request.region_index === record.region_index; + })) fail('Invalid region vector evidence'); + page.regions.add(record.region_index); + page.shapes += value.shape_count; + // A projection may retain only a subset of the page's candidate regions. + if (page.regions.size > page.value.candidate_region_count || + page.shapes + classified.reduce((sum, key) => sum + page.value[key], 0) > page.value.shape_count) fail('Inconsistent vector candidate counts'); + } + } const sourceMap = []; const texts = []; let position = 0; diff --git a/integrations/javascript/extract.test.mjs b/integrations/javascript/extract.test.mjs index 75299b4..a8c6426 100644 --- a/integrations/javascript/extract.test.mjs +++ b/integrations/javascript/extract.test.mjs @@ -85,6 +85,127 @@ function scopeFixture() { return records; } +function coordinateFixture() { + const records = scopeFixture(); + for (const record of records) record.schema_version = '0.18.0'; + records[7].page_geometry = { + coordinate_space: 'pdf_user_space', user_unit: 2, rotation_degrees: 0, + media_box: { x0: 0, y0: 0, x1: 100, y1: 100 }, crop_box: { x0: 0, y0: 0, x1: 100, y1: 100 }, + display: { box: 'crop_box', origin: 'top_left', unit: 'point', width: 200, height: 200, + pdf_to_display: [2, 0, 0, -2, 0, 200], display_to_pdf: [0.5, 0, 0, -0.5, 0, 100] }, + }; + Object.assign(records[9].config, { dpi: 72, max_image_pixels: 4096 }); + Object.assign(records[9].crop, { dpi: 72, pixel_width: 100, pixel_height: 20 }); + records[9].execution = { pixel_width: 100, pixel_height: 20 }; + records[9].raster_transform = { pixel_to_pdf: [0.5, 0, 0, -0.5, 10, 30], rasterizer_dpi: 144, pixel_width: 100, pixel_height: 20 }; + records[8].crop_geometry = null; + return records; +} + +test('0.18 coordinate evidence preserves source text paint and content scope', () => { + const records = coordinateFixture(); + const document = parse(records); + assert.equal(document.sourceText, parse(scopeFixture()).sourceText); + assert.deepEqual(document.spans[0].paint, records[1].paint); + assert.deepEqual(document.spans[0].content_scope, records[1].content_scope); +}); + +test('0.18 accepts explicit unavailable page geometry and null preflight raster metadata', () => { + const records = coordinateFixture(); + records[7].page_geometry = null; + Object.assign(records[9], { raster_transform: null, attempt_status: 'not_invoked', selected: false, execution: { pixel_width: 0, pixel_height: 0 } }); + Object.assign(records[9].crop, { pixel_width: 0, pixel_height: 0, image_sha256: null, image_byte_length: null }); + Object.assign(records[10], { status: 'not_invoked', selected_attempt_id: null }); + assert.ok(parse(records)); +}); + +test('0.18 accepts valid large-origin regional raster transforms', () => { + const records = coordinateFixture(); + records[7].page_geometry = { + coordinate_space: 'pdf_user_space', user_unit: 1, rotation_degrees: 0, + media_box: { x0: 10000, y0: 10000, x1: 10100, y1: 10200 }, crop_box: { x0: 10000, y0: 10000, x1: 10100, y1: 10200 }, + display: { box: 'crop_box', origin: 'top_left', unit: 'point', width: 100, height: 200, + pdf_to_display: [1, 0, 0, -1, -10000, 10200], display_to_pdf: [1, 0, 0, -1, 10000, 10200] }, + }; + records[9].config.dpi = records[9].crop.dpi = 200; + Object.assign(records[9].crop, { pixel_width: 17, pixel_height: 19, bbox: { x0: 10000, y0: 10193.16, x1: 10006.12, y1: 10200 } }); + records[9].execution = { pixel_width: 17, pixel_height: 19 }; + records[9].raster_transform = { pixel_to_pdf: [0.36, 0, 0, -0.36, 10000, 10200], rasterizer_dpi: 200, pixel_width: 17, pixel_height: 19 }; + assert.ok(parse(records)); +}); + +for (const [name, mutate] of [ + ['missing page geometry', r => { delete r[7].page_geometry; }], + ['missing request crop geometry', r => { delete r[8].crop_geometry; }], + ['empty MediaBox', r => { r[7].page_geometry.media_box.x1 = 0; }], + ['CropBox outside MediaBox', r => { r[7].page_geometry.crop_box.x1 = 101; }], + ['nonquarter rotation', r => { r[7].page_geometry.rotation_degrees = 45; }], + ['invalid UserUnit', r => { r[7].page_geometry.user_unit = 0; }], + ['excessive UserUnit', r => { r[7].page_geometry.user_unit = 75001; }], + ['forged display size', r => { r[7].page_geometry.display.width = 199; }], + ['wrong display unit', r => { r[7].page_geometry.display.unit = 'pixel'; }], + ['forged canonical translation', r => { r[7].page_geometry.display.pdf_to_display[4] = 1; r[7].page_geometry.display.display_to_pdf[4] = -0.5; }], + ['forged inverse', r => { r[7].page_geometry.display.display_to_pdf[4] = 1; }], + ['missing inherited paint', r => { delete r[1].paint; }], + ['missing inherited content scope', r => { delete r[1].content_scope; }], + ['missing raster transform', r => { delete r[9].raster_transform; }], + ['null accepted raster', r => { r[9].raster_transform = null; }], + ['nonfinite raster matrix', r => { r[9].raster_transform.pixel_to_pdf[0] = Infinity; }], + ['singular raster matrix', r => { r[9].raster_transform.pixel_to_pdf[0] = 0; }], + ['zero raster dimensions', r => { r[9].raster_transform.pixel_width = 0; }], + ['fractional raster dimensions', r => { r[9].raster_transform.pixel_width = 0.5; }], + ['overflowing raster dimensions', r => { r[9].raster_transform.pixel_width = 0x80000000; }], + ['missing raster DPI', r => { r[9].raster_transform.rasterizer_dpi = null; }], + ['wrong effective DPI', r => { r[9].raster_transform.rasterizer_dpi = 72; }], + ['inconsistent accepted dimensions', r => { r[9].execution.pixel_width = 99; }], + ['inconsistent crop dimensions', r => { r[9].crop.pixel_height = 19; }], + ['forged raster translation', r => { r[9].raster_transform.pixel_to_pdf[4] = 11; }], + ['wrong raster orientation', r => { r[9].raster_transform.pixel_to_pdf = [-0.5, 0, 0, 0.5, 60, 20]; }], + ['wrong regional pixel scale', r => { r[9].raster_transform.pixel_to_pdf[0] = 0.51; r[9].crop.bbox = { ...bbox, x1: 61 }; }], +]) test(`0.18 coordinate contract rejects ${name}`, () => { + const records = coordinateFixture(); + mutate(records); + assert.throws(() => parse(records), { code: 'protocol' }); +}); + +function addFormulaCoordinates(records) { + const request = structuredClone(records[8]); + Object.assign(request, { request_id: 'formula-request', requested_kind: 'formula', region_index: 1 }); + request.provenance.artifact_id = 'formula-request'; + const attempt = structuredClone(records[9]); + Object.assign(attempt, { attempt_id: 'formula-attempt', request_id: request.request_id, specialist_kind: 'formula', region_index: 1, + config: { crop_backed: true, crop_bbox: { x0: 10.2, y0: 20.2, x1: 59.8, y1: 29.8 }, crop_dpi: 72, + crop_pixel_width: 100, crop_pixel_height: 20, crop_transform: structuredClone(records[9].raster_transform) } }); + attempt.provenance.artifact_id = 'formula-attempt'; + records.push(request, attempt); + records[0].artifact_counts.specialist_requests++; + records[0].artifact_counts.specialist_attempts++; + return attempt; +} +test('0.18 formula raster maps outward pixel bounds separately from requested bbox', () => { + const records = coordinateFixture(); + const attempt = addFormulaCoordinates(records); + assert.ok(parse(records)); + attempt.config.crop_transform = null; + attempt.config.crop_backed = false; + assert.ok(parse(records)); +}); +for (const [name, mutate] of [ + ['missing formula transform', a => { delete a.config.crop_transform; }], + ['null accepted formula transform', a => { a.config.crop_transform = null; }], + ['formula dimension mismatch', a => { a.config.crop_pixel_width = 99; }], + ['formula effective DPI mismatch', a => { a.config.crop_transform.rasterizer_dpi = 72; }], + ['formula bbox missing', a => { a.config.crop_bbox = null; }], + ['formula bbox outside raster', a => { a.config.crop_bbox.x0 = 9; }], + ['formula bbox exceeds outward rounding', a => { a.config.crop_bbox.x0 = 11; }], + ['nonfinite request crop geometry', (a, r) => { r[11].crop_geometry = { ...a.config.crop_transform, rasterizer_dpi: null }; }], +]) test(`0.18 coordinate contract rejects ${name}`, () => { + const records = coordinateFixture(); + const attempt = addFormulaCoordinates(records); + mutate(attempt, records); + assert.throws(() => parse(records), { code: 'protocol' }); +}); + test('content scopes retain page roots and repeated Form identities', () => { const records = scopeFixture(); records[1].content_scope = { page_index: 0, stream_object_num: 42, stream_generation: 2, invocation_index: 1 }; diff --git a/integrations/javascript/fixtures/quarter-scan.jpg b/integrations/javascript/fixtures/quarter-scan.jpg new file mode 100644 index 0000000000000000000000000000000000000000..f0df8c188bee3202888a3569c29084bb12dae9bd GIT binary patch literal 38165 zcmeFZcT^MKyEhu7O9{OSQ2`M_ktQM~DoPO%#e#?s73pGBM54rmA|Oatzy?u4QCg&f zAfZ<=D$nhdG=F3PuZ-DtT0h> z`y8AdP(nf|lo0rXS{X;#qgDz1{rtVJ`uhV({(j;O8h2c~OORCcDHG z51kfYe^JRaI^(f~&hGLSW%u`xuGyJOF_NoQRMpfq^fu^kG}vTrvCVS3)eieTdmS8| zocA3*a`c$TanBRJXZ`#G0)v9XE{9)X z^A|-Gl~vU>wRQEcTHD$?I=i}idOr+&92^=R869IyPEF5z`TA}4J7@9duO;p>{QJ+} zeF>q2|Ftb}|F`>+2m4y}R|evL_a(F{0$jrKBBJX|#1wWN5`PkNNu9);2! zEaD$uLB+<~SrB~%jacRrvM8Au%~)j<7V+rYgvI@FPni0A_N{&2x%~q-Pp-0j`ub_f zvj2wPx)HH9om+)|Eq5BjJ1h~2*HC1~0Bo{?3Yw!Tej0mo=G7}GB94kLm5**2zoc{g znGf$tfcp+bpCz6gxy6gd%p*gGJpDh=&N9XhJ8{ISLxk?nwOOXS7Jkogu={z-t02`` zp716kVOK`>vvSrxjL3@5G)qY$KmuLBR&b^PVdxQ4I9u^(14YZ3@fZaDp6f ze7b6Q3zntz@$y2Y>n7EdD-5qzdY1d8=%_B;qc~V^flUblPl*eno773e&VDkb3%7`N zvJC;}BvKhiYO~mQ-<{6uO~OPPOv7Hjunk$HBhpbTsM}#MX9eY3ld*y-2soqm*Mh$fF5}q&$R0q+S#{Uc`1> z7CipK$*V3pIpd$RzrHTzT;3(SU!hkEOJY9;_m50QVT)J)j^z%mpkm&D9c;rgb7@My zz`rOp#_V+~C<$`nk{i0CbYzwSCCq6gJUFfUv-$Sb>}_EV_1R6SW}-H^H4+uz zTILEWjUmR@=g?V8U&qXi)d}bOyxxE5?i=^sdAsv~HTz=5$QPOtnOUj523o#FW6n`V z(QQ74?@E?zw=Fm4X(#=3`xxpTsBd37H|mkna!k8jekbPq>Vwf zUR4tfNlPH<#tAK7HD^AG%ru-g9$i6kGhtEQQNC3zT+g#4;-D2&N~2}?x>Q?S+x>?T zCPqa3`y_TUIr_M^zTn0R>I_yRFh^UH1LO}ixW8<;*>y1}lBu=BY1)13 zE-?-MMZgOOSOX2~)A?GWoH;~t1;yL$Y9 z?e*8I>9tol5Gee^4NkH~Bo3{h9Q#*Lcl+8`Q2dg7WPK`FREhllC;5vh2fKnYG@PTr zzKl^FBqfAJN90v7e1DoUjTwUzwUZ&x5vf7m71**9BWYt&^z9EexpCiphpZQY+kTE8 zFCsi+DW6V!SF&^M>Q&KECLf}(`>}p>*w}-w5XpTHvmcZCF_n&NU|N#KDUB{%>b$EW zBIkXD&c+5*XMe{typMOR4vm+@eYu!hr+!8V8;t$m9^L}8%aDFd747fC62mEt7F-+pTZ8HVMSAO!n<^j_8 zV%AQjs-`aCl2T}f!9 zXHxRA3)_vidIjaSNS%t2oWN}PBXRcJ-Z0_wtB))+>&6R*AZp88rxjHG2{;Vc`QC&T zlzLfZGexFJi@4>3F3_YlW7M}Mt(W(wKe_rn4mvp6`cwI+sZ!!mm#jUaH{UABwno|Q zfGfc+w8<@Mh&b^u$FjvO30LFB#U=7FlTx1*+ph)i4e9c3To<16gw#!9_vPhO*UZkj zdiAXGEM2)^Ve0rupex!#c2L;U5Ctbs=XW_zFPf_<0p44r0zWW@_UKt_D$$R zJ24U~sP;n)Xm{ElJ3PPBM}Gx1oWMeybFk+cUZD3hGT71VD}pEwW__6f4cek&+h zz;RP@AYG0?=Hemc6hT^Zn(8M{pHbMZ*P!%^s|q`FPxwvK9WO<;u8k2=UexDn`0;G{ z!tm`Cl*sKWj&VCK8j%}b(&4C{xcOaUIKK3J;l?J1!Jm354}|rFx2r!wH<3@L!JDx2 zfE>|{fL(=%THzFhzkve92@((mJp3%&)q0+Mj^_xx-4rs0HY1M)@E+96|1Pf&i{N5_ zY%`G#3IyDZY{!m3~{cDPX8hn%@MYLj3Yk!mR;QeI#+DhJm{&5?+o+|0{Lh@J52U@PA&E%GJ%;I^n zHg^8X2kfuZV3eSnDkDfEGYe?t0qmcX*fj(f?tYjKJdXckXo5R2(3uTfcYXauo(k6V8&&wpvuSF7=Jt`_n`y^f@&JG5(~Me z!TR_PTP(AXeT(NrZpBPqKitL%xw0dGzwS@x<1Zu5HjwUwU&_PxWqvieSF)wIosC+3 zdwPt_Un2nAH~#qJL+?0uHrmHQ=)@F@k+^Q>^$FMQ8?~pC;9H2435$* zOqd*+zf6bims0HeUuIrqIyd%dTIHYY3*EssQajg!Qr2HwKqNJpfmkL53td+2qQXFn z=2=%GR9kZow}tt*um&5hmX@GL=EDtrvvsp;L+V47?`Dr1NIP4Vg(0$9f;<~#-U*=O z-L%S_`F@V_7gyCToW!qO4J0xw@s#HhUT*C@E8e#b@377z0ZAyqJbaOZZo#RNFo9%c zk|rUQaU5DmynHM4@b1A-$wjKs*F?W~zix5q)DM+Z4HX;S^}f z0j2<{o)tXD+fZ&YDr;|^zr8y(K$j6{e92zz)+Nqfo9IEhnC9H6d?_DW;2v*~dpb#N z46ZdHVFl%4W4RMkuT1BA!cUoQ(F`dYR}OVDyFrk{N)v}k%U&GqLat{3PcNno5S9`g7ODuXe=<9I4BO3{Xe=7NB&+dKqxTf z{4Xo0H-3|3VcE`$xKTmyJN(dhMe(Bb9D9-fw z=u6WJX3_TrrYop;TR|rF+fdV*6%iW<&+;kp4^VWmzzw9z4k?9m0$}dV{{misfQCG zDb`G3#$$qUn~&6c^P9a;eiE=oZzq z8JU2{NhM9gkG`*+uuW(Blik|@vxG<)G0-wh6X1ZR#7cab(`k1>4Svl^XF!evaAqglrd zy42SItZ$E~048++!SBT-+v&u%HpvN^Y%D?}F0j_)6$Ag|C5kyWzxk6=B5(ZhCAG5Z zx~~&Q_I4=sn?`B%QVRQ(TR>#Mjy`G5cIgV87whiCNStz7JM9sJyu(5Ju&}-=8_Tq2 zB+yhrYK2b7xl)^M9zrf8oEYmw-esh4m#=)gy>qJUP}hM=+o^A`uufq+BLxdNF<>J~ zGM0G*quiwP-A5ljJ?W%Bh)6Sn7tcqo3O#{X*f;OV{=8*`>xJzS1(XE!;3S5iQ+ZWFTmGk5Po+F>&=kHsD4@d2*vu$-#Sb zhm(G8xHUz+aqU0~*|xs@rO}_-n{xXf3Vxj_tDo9r)`22%iQC}>CiNB#U4>1e$uv6R zO9&C|ulClwcpO}F9laX!!dQ#&u-=E6;zEXUnz%cB#3TwWRfLGLu7T} z!xTT->H2&+LD<3opu_q+)ZteKd@pen+$Bt&1_$t-rs%3hHT|<|Fi{q~FDV;YCanxy72y%=O3!#8yYJ4GO+pTWVhwLY4~R{`|fpzmqgq zU0vh^<&&j)-+v<@9vjg zswY!sHLdOq>ljEy?ONZo4bTuyW8S8rhq>edfLs74XbGp`G)~u)1M!gRO=8e0Mo+=< zlUiIV>h4z0FKOr4TZOXpoN|JT@hC1jngWF_@a1!+sL7>#aX;R{K33%1=#68S;PJMI z`=JHN2gK$IlUpQ~v)+Xb+juaBJZi3`JUkIkiBRJHY_Hg5V=Cq>AAd9}K}2qsNH|qQ z@cOTvU@LHZB{$}=O2})9-kPEkv|v8F?p@ zPTC8$fuplOLzP?{=j%L}w_ZIHWjn_OF)C(>+@eoAvyeKrF#jH81A`POg)rWR^FzkpV_BlZc)Nb? zt>a~n1xuyLFPpvvBqAQ$KgY(4pt1kXNMqY3L?Fj@JZ$SbiI2BiN3h&CT-9Tv8N3)Y z_BL1fqjVpWy|Gwr=9v4g=`|N|m6-ovu4j-AY%O&@7nrO9?D?D+B1mmAB)6Vsr5Lv> ze9IZ5aFr3t46aK=vC+=^mt6Nq(igE$N-r2u&eGy9j~PeXX%ct9-UZG;o8oO+{x76pC^--IS7lxKS%E^q|mQrT_fG#s4{GPKIsihwThoS5UzO zJu*wfH2Hj2?Bmasw&ahWFaN^ZUQe=l*~y2T7_& z#K#KyoNV+HH!J6&`{H7`U678XMgQyOow%Od&}V&YkW`x7 ztIZ8WC64Z)dlFrg4Y{Vftrosi5tciyQs#jp6PE|y+W;uS$EjfCsMrv)1m9-xY?a3l zQIDhPmEhbsP&l4*!=lCM$4;xZ*0&`l?`9moe&wqcB2p>b1n6tB zopRmaWO!`@r;N7c&hZQHq*`yJ9r~)<_FaZJ2qNN6a?dLw0brnEki37eDqd>iD!@XJ z7uQpeif9p0g4`vZH?E~g9eVw`DY4A+WS9B1qL)8)o6zzbzg+ck?%92$s&LI2ni7_I z2ZONzh*|6lxy9MelGIGD9Ug1cfq0V}ip8EnUnjLb-=P-vNh^HXT=M2y;qu{K7hdS3 zv`Ut8?cUel5&k1VZNq~MhB9W%W$7RZI5c@SAkpv#X51c;T?Y){B&nAsR82{2LJLY7 z(Hu;B(6jCdbtod5XfOLB?D@nUyCd!o{KKo$u1NJ5co%63JqWg>GoT>MJeE_*B?#^` ztwjbIRp^AWU)Lk2Xo;IGY_YDis1EC(E5#R1H4sLGF}a&(28uG?oUvT9{gJFDf;JMA zeub#>bcn%4Ls>He-^q&nLjvlgp{uE)`B&$G-rpC0?@U4bK9)Ph>TEvSrc-*A&IZ_0 z6qNB|Lipj}?nB5yB&(U!ksR**fnWDXY8@K89h~Xd$F;0&}~qN#zc*C{KzH5fO%+A`>lc6@Zu7 zfbwW4M7xvx>CC%vA} zZ#Kc{ky~2+6tX3s04?pqRBkEeZ6t)XjnUc(cItt~Otx7^>fEzW6Dz3Dxp4(%m+mFh^2Pq(uldQr>(#b!vyG%iSff?H?mD zc6RMcLsl<<8eK4{3&+)k!S>v)2~gKoP#p_n7({j%T5j_W@cuya#g7&_mm2BbF}~sd zs{UitQQZfG^Wpxfo~WK9n>ubbAJa8>3KXm`5LYqWNsC`WDUi?Ke5ZjDl>W`EEo+fuXaS8je}#iZp5`d? zGH~Ga5FXDj!SKlB5%wfm;Id_-@bY?EuG<){jZuqE!ZH^yNq^k2AX&{ zq*2(u-T(G}mqRR>jh_r9PUY6#VUg_V7jf^s$yk!LppYblbW@S^B~+F3kbC78@Fq`toBP;D?-?~BUw!r1o7vZg zYKs;<@-_-XQe1HknXk0m0$AgX8NUK5z1N{iRzgyvqh&ioxiGP`Y2(S0j-jGCeBEU| zZAH<+def_S#f{pu8is2@vjrQn09!S%0_M|7g0ag z8pH=2^-5gW7xiS?e7EXeA>T!j;BAknMInn*h>R%QrQ7 z9PUfphOBUmj!S@*)jh_qk@b;rgN z)IIhMW4UqHvw#~p1OCgC^nt;}BOrqv(3&BJ0QCoj%U*3GP?~reLgDR>!n%h*hscQwB7mAk5>3E7=anVIMjzPZ6j{7?xQ;(WITij?b}uquGx^ zfRh8RV++;~7d1w2XNV(1KH_lNr1`*Eyfkdx+_Wy?Nc~6rKFlTq*UBAZT@9kscghB3 zqWY^fzb?S$X5GYG>Y9V%Zy3-8(hJ`1KpJ$ZYrd}t=>@G2E|sH=jz1vW$W zw%3{TXiCtn+FmSEpOK77?*Ka@XbDXkb31-y@ia;O?yfCF;`g=~dN++af*zw`x9 zi#&3h#Sqxe{Sq1ji;x>7Cl}xig&;$NJP@e?$Z}VnmloEe7NT^rx3yddmy(o!lIo!) z;%c6^2J z*v;796!BI54PvQS2H%%Mjw1>2eZl*K__zl>oQQb=XUF2lkp}tcF>hDX9u46h$?Kul z+S7i+`VB4P%UE{%v;S3lZ`6|kxcThvdJc(RI!odnNusM zaURlz2k!B^=S3VD?DsWjKgS!7E+g%?LCzbY{M!S(`>vpt=7%1M1op%WCv*sp75-1U*pjJ=q&*_rBC%KR}9z0iTvD&dB-NLA-6c*A#@>VQ9;fxH(}#@&x1%3 zA5tq`N>}`L56QXoqwDEock(FpB{q->TYjc#qep(X{T(0SM6HqFRJ+&}aJT=U&<_IHF)j<0cA?XQaPC^}dtWuHc`ebhJLbslI-fFIPumZ0DS zUrXr3-0(5txp5=-Ybj6>`@Ao23#0maE!*dMZd1hhszvSsFEaqI&L0I2HGn8MMkyZ_ zB-&|#*f>GMJ;zky`N{xp_w#W41GkuTkSL!Z;5xN8=DiK6tMWE@W@c`JI(xCnLhYc# zRcarw_WvLFP;Neg-UbFh$jsY$S`VB(wsBWb8o#bum_ABhyW`69e7o5Gz*oy1q&T2c zwo5X?#{|)cL~x#R;M;o_p@cT~piZOEMcJHlht=4%%F#g0>;>_^Fz`mkydg)JyLZS! z;@p--m#2o`Gzz}$J06_9xVfXCzcx1}@ZI(`cSpOf@j-Nn(FTkA+vtO?g*=U!!GJtM z8q#i%Y~Z8e8**8p-)6laOT=mFO!|>-YvRW0u z4N$uWbtgZ3^ogcINSK=lQ9CG!C*p36!e8)KTif}KS6JI8w)$oSPo}!(%LCd=G?1e1riMdr6Q4`r|J*)e$x8+ z_WOmy>~(5}`KwWUuHY%(mUjWpM`SrLwE&P{!;fH+Wr;3$nc4o+MMV4fAoKQWZMhFA ztuNyx$FCkxowiw4Ubf^LU|{P9NSYMa;qUfA`KtHYZ{aR-)j~6NR1rbz!B07b+&eI( zkYL|B)TC5VQ#VxYKB<(A6Q6a_&`Rnu`Xp-z=1Bgn^(?!M(Ch_ejaEE!IK)BF(CEw3 zB2QLf&w=u7@i9o({aKbBUOxoC)5xjiOB)zoc?SnzhYgE0fldn%it={2g zhqYK(t#7mIGPSHb?u)+WAMAFp>2B|y@B+ofy3=#sua$&;2DDOPom##Eok_WMiu1J% zJEFrPQA$p{(X0G;7AjFS9qX30Zx2AvY@P3kvDWx=AZerUP1u5Giu5xo9|SM(Y!P~s z1hj7Whr*PPoeY$lmm_$(>HN4}*J!?*Rs3?Bozuj~_mJ8rNhnrhIM0OCPgWx7eZy(k zZ6MKo#5k(S#4Wyd>Ky+DmZ9})Hz$*BzvQvcE}Aa*n%&-4Oj(Ji>E;9)_q5=C(^|gQ z`z9=5nR^x5RzF3PZCPW7;gnIqbGf+vsg{{eA0`%^lHV0IR=m1#L3rQvr7qikC6tFW ztx|mcBSmwZd*5yJYjfMA{jTZSc6pW|mfeQAJ$=q{!h=%Vv&snH*-~4m{OSEp(F+VT z*&j1n%)+!A%MrccLp&Q;vrXHOb9S+RlBx(twjR53BEC|$;$A^y!`(BUThwPZIGCEL zTsl7ea5!u)N_h%26KKpl#!KTaj246F4iEFtL<}bZ%tjgV+}cS7aM06W)n(F2XK7Ms zJ9k0%!TQwg!(m!QN%<%tpO(ESf;fLi1R@C%CZ_=b1t5_AZ)E;0pESsxdsgs(?C-sa zw_EUl7C;x~+m#P)QMta4-i#<#emK*RNOVVp(m`<KE6@F`uZ~H8UaE6{g!!SJ zYb;xEqp{pz!2`u3x47YgJfaL!d05}Mx*GDhQS$5YbEgM2HY+H?pjQpUk+k?iuq_KOOt^mjY`wQs3xcT`Flwi#EJ}oCC`O+0E599m|9jx0 zRds0q235Mvb9jdc&O~vZ6HykTr2e{`-Xohk*}rN6*?3k*`|O#s_{{(Uo!q)qHj#w- zUCHv`%a0?H0gzkAOTteUWnLblxeaahR~Y?nN;oEUFZ8VVx=FKy#k*I|wrHhX*Waro z)b`)e!-L}gB|TL7@4O4;ahtzW5JNjnxrJ4D3e!1m+^w+NBwmefuIM^Rmm}^NFb=oU zjXSjN%B%Zgcd9}(Hf4?;OcFyW&wA9nf1!Y2f`KibZ^7e(;cU zSO_`9RBfjAzJZYQnalnT4x*mNVv)3J;jL|*#LSucD^3k0_JHf**%M`S@yVv5Cs9h^O~(Q{j6ru$%5FQXz+>S-%`Fd@Y`n!^lWog0 zc>n05(H4Hadmw-O>ienNXUn_R|2SNoar{(N$fGAwCQaIWD>xELYq>^lBgff^yl*sN zecpi9-}jDa`?SN)&rjRsT$035-;_(vtx=}V3qFS?>F@?nFvE8>ZTiR$p8^vY^kzYb z*HE=2btyOC80?n+=`-&0{x8nwH)OkMl^8TAehWi zWSJqC1ArZdfJ8#8sxH&Kns9{;~F%^ICa1vN+3M@NZX z>i|*UKIuKVoI3x2=N)wA50;JPZ6_Fay&R1(^6IeDs%^|3(AV$D+-s~7Ie$8DY4-9# z@mS*Dbo?YhGdswB6s3+S>L`zE1c$Lwu6?cyp&p0Nh&9#P@{1FdpWd?NzU+_e2YgQB zyMu=!NzJr1#E9!0mnq!5Eky3r)*rN?va#}TA?sPzbKK|O7j;d3*D8)TW}ZvhKIQ%9 zO3|idl;|mPTOk;Y#{Ejc0?-ej!bHLfs=IpO*Fkag|AgZ|v{O{+e_j9I^1$N`-$77E zJ4?HPt8$E|ouwz@w5n_|li1|Y)Adl-4P(8F5zjl!EA2HqSEbuK<4uz@-uOq^o=AFH zN&qd8H9TX!?hZj#!Z=moLm#`=S#@j+#(KSLJN`;bYx3>bXpg}E`pK2XLT=_iG>xpF zPC(QIY!D@N3=@lQyclj#dxKq_=!T_&F5!B7+j1z~< zlNp8=cIA-6;j&A_*_IF4++Npd2XyD^+SZ{kc9kse+uDh*%B5E8KcIhsrF(V42tJ^KaljKmRzJknPw6=MG=qh;B zWK6)f*lE|MJ8r0+%njDm#BYr`w)LlJd|2!ab>CDWs?pz?lIRKWzEEVK$ufaOX(o>c zu!e8)_g>~1RT;V|?9Pb@6^yJ;FMbm_xwUJ}FKP<;`~s{?Z>NrXA**Mfl7_I*L53E* z2{&rYq9tTvrC`USoO3l#yeiM7=uGVH&RG9?s=@kdL635Ye2vE@I}^Z>J9hHq7A2wy zPd5;&2960dZG#nAE2vv}N}SIz_P&sB$B82z1IH`&>jrH$|6Q^FQi*~j7r*=NCrb?! z5+Cw~4zk!5&=g>1W2Bnax>a|G#5E9Ka(^8)z>tiG$~$6Q>jpZ{3h;qoqiLS_61C-I3kj5JXt;t zNG%f^1XK$%iJNDz1iq}=SfWAcCm7di8DczodG_>RRp1!{#HuL&v_D^$v+fc4+k;k; z(LW&n8DXxuX~VxjK8yCgI(Ng;KOldXu7_No`d^S=R#s4KxJpv2QFZbwUzv_bCPF%0 zm3(8kl6Qq~`o3{jkW~!N`9a4)XN!lA(tRH~^iOZq$+{h+H8)9vt!{JMIcUD(ZA5Yg zn&O%X(k;Bms0pG|!>yJvmB&HEP{p0lOkAVM{`9v8PPO&~iQhlquAd&GCMGQalZ$7u zutgXb4+}jbYH_e#ZllJNVrAwZ)7nX99BMlr&G@d7J9Yxp&D9CMQ)$&bt@P}=?!_?&-G21g!A;=W~@rM&N#GTq*-vo>$ad+FLUFeZ3$B}`CV%!z#J%ba@;x;cR1BCbNj@^TjxCkhBod`?(CM@!@N|8 z0S*vy!ro$tVNYXVsN}V(~ zd+=NKaQfEGucvq>>6H!z5CyvUwH2?)j|9R@CHvDr6$@>1^*ZbN;ZBTo!k@Ylx|Z}aB5mw?kWmXhY_ZoHy#z{-h<;Nv2obSVfH!}S_6)4bt;{- zwQSq*(Ie%I)ZIn4`X*BNrV9roT3V<_sM6rL5tu?Q*9YTHZNhDoihy~+V}j4s%kcb? zEy<8zV3R_;{LvEUdB~5@7#KewhEJ(&uG=-NDJ*Hw!?UAXsA`QctH$4;ruHk zBz?d8ce5Q{o+VM24u0yFIAt~d@SqrtZOkJf?Tqrqp&?wgKHHT^7WVP!_%b@%w3Z{X zb>gIBrcHb8C%?sixqg~uKFpD1|SoIa4+=>OZ408pgJvwSy%7bw}G3@!# z%4+`;`eLUa|woL#$NKy0`-A-aQx9e3KoNYw@H}2DAs00$v8jrW9Gmd?hJFA`D7%g?$!mlu&c zg9nJdcYJ=lG7bLyd3wrrd1`{T9m}lIHiQ4hog*L)d?aoGcVz-@Ql*eZkH6;kx~ZOX zWcKj6OvJ6CsetvTSYrNza@5vMm7no`p)b?*+-n46c;D&MZ;=|g4p0WW4>#l3Pn{d{)Wz+Ba2 zYj4k*oBLj+?~;}j@vs5|E77nr&jRop)cUK8JF(IPon5JJM&A*`;k7SogCukvsAd}j z)~;FGx8{NjX&Zpx5k2@c*ReW>|f94@t0YJImF+key1)|WrYu!TLc8qJC1b_kMeTp(j^ z;xIlOr%6=ci0sh)Q})y(^T*fy_bs=@utqW$J0b7g#~0nzdh}`)gt(vSd?gAZ`Bl8N z6gPDTQ;TW!qKkjCd^rkW)_~pG+2Jc!a@>o}k1gmbZdcrquXJnS<*~JEG=5Ryu&Q?I z{EKi<0nrzBoC-n95xj8aUj3Mp1J4-vaG|%l$9^#dTf=buwS0}gqg!1tfH$JTjc~#g zZtQI!+?Y7hNj( z-i&#iXAC6L3nXGd6rW0f!$7duh;6;)?PKspRSvE0<*d~h#Vn}QQG0TVux=!DZEUi~ z%mQCozXMVHTbtFw@l*%{&3PLKs$Xj!hS2dp%s+cMy-xiy_CsZ8X|ZCj(b<9|M+f&^ z)OWO4281uHN`sh2o7r{n%UpE=79p>1!6OKe8GQ@!M>f^V>t zccMrZa35E?0?>E?(D;s4*)>1Rq^JbHN{I3u`jzX&%8f6-Hmnf%%J%aNpL(k7)7_~i zUZjCd#)dC?_wGiK>d7s@q)*de{VsM&Azychj87Xs#$fgr#@HMVyUOY-rfbdq(Y@A^ z?p59Ks(KFPw@MBn(83H>drp3ORIYV1Bbr zkGmRwJn^$m8nMDrvnRQN$m@|8%I)zA ztLQona?gu|^b%1*w{aE#Yqx}8nVC6xX4*ovK=O%1y&Z{^9(SVOxy`q}s;TNV#?r6D zR8X#u5s+K3+btpy9T9qWdry0oX8TQGf2omXardcJwv-(qFxO zuJPS>P29mA<$9~}ERNfFK6fLqpWsFePX?Z2szeu)`h8S5{W{P3Xo_nP`Cx-h}Fd6=TecbT%)KL8Eo7W5PaYp-ki zcqYDctaE!ZHC*>ct9hKS=$p8hIMKu|3z+lwY-Gzd3mR~REf-0)d?mqCEb}o%_O#%k zxiLDpf^YJAa0!PF`m?7t==p5oxy?&QohH~RjkR${=o@BaN9>Lpj*(jm^ETEAO0mo& zZCzNAvxH7;G%Fx{^?|HDEIF%x2vxYgqj%C}gVpC5aQLqCG76KWwy{|lqfTxy2ZIjS zmkS-Br)j0br3f+*J)|GO4B2TCEL-n?rpNJhu2cwe4rFV2ImGLy$!Z#EwfCC3DS*5p zu~C1Lz8YR`eHD<}C^Z?1J@MY6)J{{N>G=Wg{$&UmI;*hvqI7v?X8X6^@89=Tyj+_e z(n8uoZUJ>5=3UAvn@B8}y1j+1#kXzNY*)E`*SP9l<57{@p1qP{R_*5|>{nTNZp)Zx zG6XWYhdt-^cV@^6%gn>cVtuir!bfW%nXm1)>Ke5k2dm_CzBfn99`|Zf>XDxQB5jnN zezjm`{Z)x|QK#smZd`k?ga_6ioV(NrXkjRV0$wJvHh9X_iSIat)!;io0_5?F&snQH zKRB0pmMYa;DbQOb9d30-=uaB7FzMDtyEP$G4vs%jx?Wi(^07sGzs$Z`NT_zAp8GMd z{qvf4l{(2WrjpYh*1TQFM{;GtJcT(8M}z74Fo+s$9^_}@#efUF;`_Jg3EQ3A?I7mS zWi6-DwVeunwSX?BV<8svGL5hLvy%)Rz=L_u8}Dr*dDg^J@AQX>n!YT$ntkcWxBDgA zHWN5q7nLeDUUmuTR#Tba2<~=!=DQPc@FsF1Nu&?-IbGlYS6~O#)BIHfjfQ7zE|OYevG%fcf|1SI~U&5b12dvIAIJO!jBy0b`Acpu>#f1A#E{EO>vOq z{VB)y7I5CudabkDrW3EM+G$*S-t7c8iJvT}#4=+sVzi*NMBHlfd7w7y0@@ijD61Qf zG6iQpe+tQyYt-2_D8J1Htr3){yRJnxwWz3~!`ko%YPA4gY3ct#hm=wB#o}kSik|RP zdO?)a+Y3%gLOuKt!DEsTsh^}=%Nb^?bGlj()cY+|m<&?Ku3)f_q@~$K8f57>ES<+ieMzv)$J(ce*qDa8E)*$O*Li)ZvyY)6^ zPwKWaD#sDR2y;ie$J!+R1ki^;+Kh^i5a@P9eW3m7D)VR{x2ThuX^aX+GiCHEtCk(a z!L#}kQk$#^`EAz%62s{?Ck$`;Ow2AvsAM!=;rjR=`(RDDcgkL&SUK%>@}{WkmH$Qb zbydHG1{iPI@W#+Tv-M_zXZes%MmMcH5p=j<$J60xzE?o8V}MRf zZ)SB#na&xZwc8|78Mr(w6r8|U!b~oMJp@mIU*VTJ290jC^b7Pq9UkvuVKjB zVTq@q+UQ2RT9ff@4O3`2ub}KR0dKqC3~xz%{qjP6PfG^?wQ=Q8C88X_H+F+P{{rnB z`UnVqYzDlaTyqNPMS2(Q-g8$_)eDQH{%K_S?dQsX{sz3o?^0k-(d1rjW9i0;p~f_9I=Twy^V75N|b6% zQs9hVwx5Wig2I%o0i_blS90X42`a)}N`<-Jf?^_i>iU6RMT?<~$8+zUBb9bW`2m9? zHl|y$$8RSOPe$-H9s?9t2J7%b_~L`KVEnj8OLh|$8jc~lgtMsez+jcuhi>Tfcfwm} z{@$F{b7|3{UEcVvx?|?3^f5CHAn-nHg{mEB+VfB;SI2+r)?c@Slov64EoT5=SHpcg4Wt86gY|fpgqWXB zoZ(KoeBa}-?|;s9S~s|EQ+xMWklQI8=Ja!X_ZqpQ{+9~29@D=4PfUD;f%b#)i7)r^ zdqk6e3@EB9$FlSAY({NzqwiFxg6)Bjaqdq4kc$`8PPD3JrJPy4yC~ae=3x{Ogi>)& z;wjjTGd@{3ug9@uQri24JCHRWZ9*n>e9PbfkN7_yh(3f1a+SN^+=y*a@3zqp%u&Cf z5vf_g?N0bAxc@CGMiYexumOhs_O9?Dj&P?;X`r@v9W665RM^v{f18%e{DiNW#LUg0 zE@>BG?N97I!heX5(BN0H)Y5AGX!AN?6#-`PhElbCz$8_Cyc3XU9K)ZlTFN~#yV!%9vYYs@(1Nc~uFo+9E-e3{(d6x6Nv1`AzUnD2nlQ^tISzDc)EN)kre zoE^}Cl_460hcR7)=dro@E@5H*m5s+wna)Y?oW|4{cGH-{7+C-7JkW9330R#+Ah+2C z;-FgEBq-@7{X9ntT|uRHmKKu90sL#EuRjC~QlK!vtRT%?o*x9KH6~IZ@vgrOv|`xme2F{xsUsw-~B!A<9_bvpXZO?(P28KT-RLlUSF^Cb)M%NN*DSJ($@GU ztnA$UmA{X!S>*OJnb%IJKk%RyNI%=@sliuZ`qRgbS;>hhYn%ek}%&j9WlX%h((&Jz{iyh7JgW zIeHy>FJgKq6)BDW45j<5@W^+%QJib?8L2tJl7a>IG8OK`!&H9zlDNWkEJ-Zk2=*c> zeMy0nj|ilRcK;lr7kM;H{)S$$7tfx{($=`htR6Jbo(t_9pSMP6RsyFoFa;#A{>a%L zrYR>Yyi*+0gKWym=|_rGWT9%Kkg~x`nsrl8GtLyshcTbN^mvwVSu3<(`}=U`MNk{i zS~AR;*JD~SwzpF^M86`M0((Lhox4QMV}71$WK83; zimh&Sjm% z3?Kf|4Nt<=WAAVkp_(H~r^y?;V(e_PyI1=;E`MNG>2cvy-EHskQ45*9GPi5C!np0Z zm<5t7a8N?cR`-z_Oa{WP{5;E$>l&5t`X(Dq%ru9otH*u82xWfue^&=wdE7Z zCi9rWn}7cp?MFeX*!4BWL#~K983Ab;axED%DHydFKF5qx4GBHB3hsG^SA^n%-5lRN zZ!B~$>4`Sn;a8~dN91*LndJTBMW3g90cjSwl|+wl9zeu_jLQWpx3;IE2Y=_x6(*Mt za#CtCdR*}F_2i|A-uX*W*AxuAPrN&p?VsiN$DzIvOa4Zh#d3vXSpae(9O*Ex0LrWf z9RlRqRE#=f<#BU@Wk!Yet<342J%&Z7jy#$qm!g=4FdR!5a$mr=Y_R1USO+kGAl?|q zsqYYA#14R7wA6BZy=tigg;eX*;dA{Ygg+f^&%}Ji#gA^rs&g*l#G|_>)$)oVp{rl`mF8ofAw}U`lzUEYs-Uu4oYgUjuU;?Wb)PH zj)+p+X$*P}SGXOlpmLlN{1to_SC|p-mf~YT#>K50*>}i4pI`Kzzg=qIkfat_660d~ z`CjJZKa0v@3XD(OEkq8$Yy_Y!t4B6`Phkb3d7!{(`uAn3ZhCR^11d6Q)8P``d9PY& z+r(6Py_D_2n9uf8Uc(M@!|%kAHy}QyG6W-U%$6SK#GwIq02-xC=aN$`4yYAnESY_t zH2iw;{OQ0Tzcj~0*Pi}L*L(W3y%O}T&9;iPy+h!7bxa8JGI)q=?ifxCEn2f+(N=rM z!hHFl_6n?Lpy1q&b}N?v4=Jp#>m`Z3J##mE0x#c6^9lI$pq1Ipofc)+)65 zRN;ct{-g`R&!U8l+6VXS&ks%V&ZG(4e=RtIq~9_jyh6D`2k3%K99Gn_78Jae%qI;u zqbb-Fz(%+q*~)0P?eIU`~@veKZChm%8(<_dwxi+>-)q1S@lZ4Xz1+lz%6;l-d* zhmPg{zj^6p=}EXf}Lv!0gssL zz3v#}dy76XNny0*)WyYjr#`plE=%pU-l1kbmB3^7g~UGELXRcDs$vjKfGNcTS`1Pg z0}nBgbf`ifp>JeITw2@rXW3gb`HboCke1(ufy;J4G+90me)l8{(m|$U{o%7(T;VFp zkWL_ZDfi~}N_5mmqs`cm-Nkb{x?!b>)!RNhwuZqr%nl8A?R?&m@E>Fxm@+KE0!zLR z&~7*z(B~;S1Kp@OyDrID-Ew_rDtzba3?dkG(cPVS-lI*{BDVL!RPJg%$K+}7TfxnQeSI=t@7bm>#K za&_SmQj4>!t_9`@cs!ql59>IVpx9~7K1kPnsPnlJePBmzSa5h!*`wf=X=47f{VNah z#^O#7s`3b%*1rDw?-J<}#P{mvp~*;;tpF@#R&X(JXWX$m1>ngFOj!)FhC)Zp?mRWe&0I?Puuu|$O>lUY80{ZGVxkG*0G_s37~Tn-IM{|y|)D%d*%nP!|gOWh^RL@-rv z7yAs#?Pc3aTjhvZO%38D6>^t(NL!!WG`i4uT;m z<`Bd<{0$)8`cSy29&|#N>*#29DzD8z@GiS@Ux}!hnrd9Gx1i~EeB=o2ZasI*B;qy> zkb?f*Ow!Qe9c%FrdTz>>?6^cy)jh|tlvpEcF=Wil!GOTE(m#y>Qa>K@0S!FuZ- zZ;EejsHVo-^o4szfIxox8>rfja|e$?5<|Gc2nLmT1lI3FjR|M(+wo-S^t5E}LRq}> z>`$@62G7M~rD{}tQ;I|FjNYslSD3}p!V*k7MQ!OCOcy9jHleH$L;zEI0S_uVr)rv$ zOythuN3`x96ckEWv9;y3%LQNS1cID`t%I?pfBH^Xs-wLoj5b8jHe<^6zj9K!+a?PK zT(eZd#_SGUj`?yf{9N13w0CwVAI04kXSs04al%+X0Q6XDa>_bQp%M}h8YDKRp_)X~ zR$PNm)DH4n;Yxy$p%G737N@LKXFdVgdiN{37gYw|h0Mk_M`sy2XlrxzAm;QD8AXv- z-NC>+=jQVa#4QBErEn){sU?DKwCV;%ZHQdgMw+F{cnuJ`BB1>mGzmt#^~dkWh^Rz% z;BtkrjY8Q%muaS7KFZ@ zpPPTu-JjaQBS<~vFsS-bkcwmP1_*`v7sfJoAPl4v_Mcyn=-2%F%UQtdEO6_4LI)}` zBZnSX2$l|2d?nJ>cQ(dPp1(5TvmhqMWxR?KRx(=sEZBeJ5hfoj{UDIIi)aAubSeal z4qan?9;^~;H%j!UbZUP)SVUCKEeuNE?H{X5B@(@o&g}Yh(C^WObS9iQ&pgU0!G2x|f@7r%*N^BR>lM&ac5v zIW`#P@uirFB{<}FU5isn`ChP`N@h@M!R1cBqud!^s8?%z`DuL^_-9Z z584MzfBZi@gRN{qz_pnNf%&*;$U5%72$6m}Lz*dUNkQfn@M-OkUbinlr>rAl`LW~# zp{;y`TcmO4x4C(+T3*n!vE;*Q$GG>~G6h7gKg#zlV%-8h38KKQ6j>jg4yp#VS+_tH z5M+-rJ?jW!^pb2_yeQN;h(+Co-}~8P81f*jRJ_gjeIXxzGLO;$Fk_j=bAcTmB5)+hds`lbJX&BM9Ed@Li(JgM2F8#WIrMcEeD zE1+6(P1Qv##Rt(teEi7s_GDpkJhESRxp>?|@noe;;l=V`yo zQFvW1tkUM_+w@97XcYaUMcuiqM z(7c=qQuk;zgraO_{}wmzoqkbPTToeEmiF22d#R$?vxS&NO|g|YgdbZGJS|xEg)j)T z7Qg}e@rl?vc+Uu`$8!Jp#mPpeGb1%h<{39#^dF_YQ8CrL>Bg_H)AqJ8zCNc1C5w62 z7QG(EvU-S1<0|A3+6L;TzUa>J$ljv(wtu>cpdiz~$(oX%p&?WI_6VO>- zv6bjHtG+PKBOq8tG#2$BQ-~wIc~_7ywE6O*^KI`iF*71h?`0e4zn^6dPx9u2C=;4S zkn@2G&Yg;zvtxovO#GFYYeOE8Ntv(rDqJyA{Pq&DKcLys@nH}}v#T}lyO#~?!^!Ih z`47}ygx<2;uv;!2YOeAe>; zudv!M>)g(#4?}~h_Mom+me1nA#vfmZB?yCM0Sg_#<{09#G5VA2vt)SMU+L3!&w*!8 zY*qPVmbP7_mT6>U(U9u;RasZE@V2=9d_PLLqM&p5e>8*8DI#iv2TQO9c3s%U+6;@m zMjP_#8P)8=(^0^o>xMwbjLKeGNcG;=hSzKn#qTGM4=Lvz;%rB2F$zX|k;zK(P{Pg9 z!3w9Eug14B+t3v{p%Yzs_jB3`iI{>vQ*;m%-r?Ne2Ms+%U*$A{94-^v=vz)nbN!{G zMa89yPd}EO7OPeb-Q(b*U-hG7Ov-OZHqb2%^E}ba253!qZJO)upR z)+7N9Jf!sjPp*X19l@TD1XDTA({B9>>v#;le~)h3GpHNr!drYrCUwGUN>=5)+Fk8J zszVYFNq5}4yVH2Z<_(YaCSuD+hAYiBrf+s{>{wVBsen*6jpHu^-bMwNB`--Q?K%45 zh4+hX_AUmGc!9_3PAovmWP<+Td2K>lV>tnP9j=A%z*A_j!&#h^xNlPa{o(HO5AWSQ z=B0EoZ#u-Q^4{f+<8}%Mj96y>7xBp`f_^3k+DBTq0ZW#}Oyg)8PWIPGY&UM_FZ;YB zDpW*VlUm)BpZ$E~>GvK=W>fM7x+RycNla)U%;&+tdWq!gRE2U*v~xhzMSg;7wCkq{ z+O$dqcSw49kB3u3fZ0Flp+F1603(ipcY4lYynm` z<0?USk5J##%MX<|mCalw-oXzP9*pP++~lL+dJrk3?HKr1u>E4BV@#aso5^g%Ma9C5 zXW!h0#NR${6VndL=;FbsetMMWgyv-d0wbXbS&yR0vra8>=WyFVDZn0*V5qpY=WTdS z1B#(&n9Rx~ZY?}MOwz_Mr6{P(_L|27QG-J~e>S-=d(LyTC?MFII+ay)2~LVXf^}<7;kT-r8JjN}FRg9I(sH41RF%2@Di2 zDCrnkY|l2}fYOF&VOxXyIJ3b<4Y|XKxGM=Pca|3TWANc88mB?CbRNME&f#%-s39x( zR)H87@eE5)dgUPvu9q_E?u|x#UH-%C3m)x%4MTo|^yDp=;_HwGczw%xsLUzk@<641 zhw&-Y8QtHm9f0uL)JG{^4*0DXhT0Y?zkU1g??EE~?GVl({zrr;Ln2@#=^8XO5Fvgo zED@j5Mg3~EfK|=SJ80)0uX}xO(bIdoi}Z#4XCHkk=79vcFeAuy8L|}Tjsj>Kp7`sg z9*Ay}N-WsnZADY0kJ1JBl0lF2t@`$9GN$Exk zt+_iHmy=1t(DG1o*n+gX?a`qY5wUK!uH6AoU|GN+yO!8MDe9|*4y3G4u=F9h-omz$ z3VQo_6Uzg}*M(*KH_qZGnu+#y4ux8~KKZZ8MviLm{Z*(PS&Zh2gXJMU_{x$9tz1II2a{ zBMOq;d$7c4ateqW196#q46Z2cVw5CZFI&4&_ulVD%L9=^b_%7L-`y0d_p6kn&B6UZ z5|n_BghT2n10!&1AOQ#MH|Za7BzPLpIOA;fsi-EW9bCtUX0H;9u3Fb0_~dNl@BZW7 zpS%4FVNYOrSTaa~el%CCM*$~j=M?O(&Dxo@P1?h)bsBJA25SI}_x#V=0RI0gQqh?M z%o`Qq)RrF9PPQ&XW@Pv)hl4cB7R@O?2prP?o_cDL>=P%(XX1nhRvT(PPx zFO(P-_XD3Rkb5hoAdSH|m&6C-li^_ckUx`%Unt1lS_xvXKKmq;w(O(TV`9>v9USq( zh8i50{n6lpVBR510`9?>i~nO=Oc%IX4_sID4-XIUm_zV1VdqAf*BRJMBOu9RBn_nL z)^vnadM+%gO-?r#zD%8Np8l#V94Nu_G;H`w1V8V;6gwsu9x<2^kVT%=gS3J54Gb7f zl}Bb-H-6=+Ksl!Y<#z`y?6;!QUxdt}ruY1bcdNBci$Xy9bWNn>ATNgxOqWKvZv5O? zGAbPrl#^Amzh#Vlh@z+zX5;$y&0W2B83kFqD(HIS*)hT4@2gKFt!f8>&o7evB}WC4 zVazj!88?#DasG^GszBo{&9^e#d-y;1A3+RP(Lm@13^gk^cYDkIavXb$~4K-{BPFFTv;Ksi0ik zN-rHyGMlt;XH=iV;~8!^s^X~+K9={z$>#P1OQ$|_*L>!nx(_%`|8$><{GTF^ z|Ni@bXTkS>{C_fkK+Q&=1x%Y}_7zT+#Z5Y%Da%Pg!^&e)mN)trkD4RJjoU}EQ$lpQ znjKru?lre4?+&;06jF;8lU9ktB#$BQDd~XAgMih0i6T9eB*&HmckG{N+DN_X_3$bA zCRfp8KB&du&YSrG^HRHHgl18r?cnZNeY-j=K>$y#{kN>}2(8V?=|K_=x|mC{&l&d}Ao0X!G z%ug8LZ&LtyBaou8p4SOr>G&t;3*#Ljz_LoxYmOGBqkHTgq5O5uV0`eI{e>R<@zD!% ziFt;jg98(j|FKy2ZvoGHrXQP%rB`;sVj8FeOawE9t)jFCKh;TK=4jCn0WCK>Ua`Ca zXb+>qLIX{YXnjUP$`x!G6c;YY(hdQ8H=ey83Atluu{(7;4sp@|)(Py>ai3;q7jKt- zVCgh3a9RRJQk2}H6=z*wwO{`?lBxeGrP5;LaCdlaxg~~ScW`I$0tj*)whqx#jhDZI z5L~C}plv84mV@6bn$zENL%!;YcVIh87P%7<(o4vZJxSrx4HfGv=mC(J3xsiUzkr6y z^#O2OlxVF)@IO2B9h@ZEcFuYAdz9bJqk zR&M>_B-lEH1#*`cV8+w zdmwK=diIRz{f2jk_2z52!hn$>zz-VM2)$<#fkHy+D7JLw8kG%nxN>V3Q!j$nlJoC zkGVlnMveX6y|0b_ z{jGnwuXf3?uWrYJzGOOzJ*i$lgm~|)07yRVjG^T~8i?@ts~Vv+25T9Xj=oOa4kf(& zAIv82)#zmDA2*H%v_{{~$WQvP{gl=xXK3F*g2hFU7*qoKl6q;4=Li!e5d@dqKp*K) zaJXV&q*!%e)p87V)$+;A7>*uT4qPeO5`WEOZx)oyfCWK%6}E7#ur}e_k2YL?9H>cf zVK7bwJ{UQdPu3%Z`7y&dPdl#DnH#PQOx2F6C^A-dDOlh4cwwD>*{cxE-O`fw;&+8r z-KXI|C)1acj^zUaDscAD0kkw!M1=`LI54ggqge*M7}@E-i4B^?s6_DH!4+?-Z>q7> zJp1_&@%QT$djUxd#6f68ulNo(m%-DP)sm_Wt6CCf%45~&5`~wg(?v!*mt6t>YID}@=V8uZi5wI zJ5w19HvS77T%n-|EPj@}(Tr5q|CM305mUZlS5Hb?8W(5kv#o%4v2i8_*k#dylm-&AsPkw{gh1tjU`^a`Kk9kO`y~EvG z{<-$ZFPBxNt7+=d_dRIf>u(;bzw3L{Vg@K>G+;nk5Q!xmLXc|^^c6j5KWZS$ky?@1aWSX`F2q%d zNp?`}RJwf4A;{CiVqq;H>i zSwEM+e+`)S;vk(XUT7DvGMC^lUn z)%`@PgrfqlTDlqRy+_H_r>zd#&q2TuJm52lf?_N3v4YXYP;vK6Z;5JAugQ`U%U9-0 zL%vr_XSU&}MpW6y6FjTAQez>T`f&(=)slg#`?PZ=g6>9|iw2@nDOLbf(Q7FP(dd(o zPhzTjxnh&J<5MRZhPGWf__kDIZbRq$g~$TGUfV>h?wseLA@DW-;X%QO*O_jd*OvQ$ z5*SZ@0hcoh!w%dMSozREZ=jlFG^N#)F?8>{K9A{~y0-qKD#>Zsd@fycp}X&-NWvg^ ziUfCK61eaaaz11^x<}3{M_@p<-OuRvbMcPN>rnh`&;E-&H*tO)QHWk)qbx9`b-y-~ z=x0gu_%1lz*?CrrV&SU^ZP3?xik~+uBjmnq0(gNtFkw0JMK?$L2~B(bUC;1dJGIF} zRokg9JVz3kq0k2F9F}k$e0YuLK)>@H5WLCt!uT-nuslt?t5%F zJ}269$D}i5kBdJptQHB&`u?CA$^6BsM}*}-2Q=6+nm@ThpytN^6q6)DF?=JS z6?u*2W|xcH5svjTlPD;iosnf=Xv?{uqXni-aoB3GCcFl_qu}~7qlhbvqio#43RwuE z$fL4sRm3b-lC5B)Gkoi5VBq9s7^_=%?(*G2R-X3j?0qtS9s=f5FIE;Cf`m+w%zjAj z8FvvcZMh9fMo2*^y|RCcdTxh@$iMX^G?fKCx!Y%GyL>*J_Q`c{J)+-Aa0b^D$llq) ztmhSg8#tcXa$aJ|nM6EWx@MVhW1zQFgs%QJb{Qr%+aToSKCYuyeE!*>cHIY+ z>~r1?%HEoJeGZ5tXn=LZi=v~TD0)vXcBdtX_XvH6rm&)<$Ri@(fe2)#I45gkX-V92 zm#GnUO_XOcTKAk?-1>Rh2h>{IgSF{_@ebqmLvN# zc%a*WxNL}3gno8e%6NRY+=afzP zqq{pG*R)t?qd$dz)ux!I5Wc2_+6gAy!%pL-&w&l zvGK06;_#W?x>DP_W7QX43$FHIE%D6*Gy|}s9cO@}kQNzQObJdF;tERc+fJz6VWQDV z^u#3E1M?S_wD%3Y=JEC0)wUN@f5-;;vCc6HT+o|hJARJ-l0hcpMNh-Iup3a`a9jc1 zTfjjP#@<(ae)N;`7c0022|{%PITwj7x$={p`@S*O zN~E9&I?mSzeu~b0eD#@^cmJU3MWTxQ`uW~;<21G0O^F#SVFmd5r~`DHzhJCMQRIN< zi~}@Oy11r(sC!;s@q~a5?aK(y-(2sRb&#j=8-a4W3FbXk-0~2l#T8|-&IPyGrFZa^ zT_J`)D2w_0CoJ^ooBDLumNd6t0`@(jqIcrnboKA&#nrom^cX;@1Q7a>YkZ}_fN%zR zTH}XyMiB+pucPfuFB-5sSgiYc5PjicqT zPIKQW>2nI3NOLXAwb>S`Eq`(88Z(hNp_pdY$qXOa@gOH@ImZy?o2wRs=LE&?7?9Be z+A~0ahOH+N;1CEey2M49U>%AlvoAs^j2s(=WdS$~p;HiwFDmZX9J8ss*{%9u>0*f3 zxQBg>qlQEsBHAbctfy&S*0Dez)aw>~C?y*S4mVLfuAu)AzhfIhlV=qXiiGyH(B6Qu zJSiO;z99p?tXro^3r*%}%&Cq3J^aD_TK8c!`smE_U^O|DeE$8p z^F*B&jb4hr9w!rm(HiTcv=Tnd08UmW*%}J2`Gj4tyu}^>+w%e16K6<`)GObYDXa?p zsVeO>(`-{uwPK5!e6dnG%@PN`EOiKS4j4Yj5f!em2}6|m0#K2Wi3lk}Tq6_(@X*1i z>lf4waVrwgKt755PbzvlmBr!#-LdqEtQAGK(a8_d3qr)2V*uHEMC^a<=>Pam0*N;6n4F`wo@7HG8y@1LpM|< zR+sX#f;U1oaI?$IB`Adz0(wX66(Y>( z+WI$Nz3I-KAwRE}5X~R@DOF!6YQDB;{W?K7nC9s!f^q8N>#9xMT7>BKupC6g23tyt z+A^?_iKhci3lTw8XoICt3PD}t??#3aspoHRo)1B!M<+FR#?K-nae;Kux9K~HZUc_e zbA1S2>q8i@e|S#)UEu+5wx?r0;^Ej}XGoot1&jobqa#2m?036!_MU>InucYhl+TqG z7h+9LseSV|oyqLAoj3ij#?i!89{Fcu#35l8$Z!bYF+!Xn9V`um?EGth9~6dzfAtJZ zwbX*1k1zNBZAnKN?XrYJfzC7G9bYDN)b5oxzO)<(mdF^|uX;`_Zo?UD%;fhtaGqBV zew}QDk%BF1elGX(K`m)MxrD~MUu#1E8ix(>VH{FgW zQv&R4ub#3J|GRYu98lr_XaXD%3(VtbQPeQar6G4lcpNadD`Z}AgVRL49=xn6Hc~Cf zs%S1Ka8PrN8|-&3!4kUn!XXtHZJW?PJP!^4Yv83fwUE&eFd?QJa6h9s4!^JPu%v){ z#nTR6mOADG*N{`rJ>cyCPhrC-L$CbKp6AgYzT59@v&(e5!!GS0m|K+0A32r`cW{ey zt)NCW$IX9X&U4y3)zE5?2i51ycfDkUu;q>rVlOu#O7;C5KWoO<_-cm1$-k*A%8D#WO_&e;QwL%$-56KJqPXUPC z;7ZsK9Hc(l4G4l*aRj*q*_ab&_}K4cTQ-6pGU#`|8{DQ{xH%*H!>ebH&$Irg)PSzj zn++sG-YICJAJJ^WmcPx4Vk^_J*`QX8*n01J@M|>*?wE%rInzE7NKHWq8Lg=BgHtF2b z3awDN3g@3UoGu0X#Z9xg0^isxgpt4yyKz zgl;~CemK%1eXdB{FTK%{R`$qlFYBZOUX^*^Lrl8YG9S};gG3cvRUXDoe)2M$CdC-CIAIM~15M*Vb?e&nq4 zxuRaUG$Lr;f2wVNV6^5Ctwa7lP54GJ1%FL7Jm3+M)BOTeCgf zJeqfoC|sQ%)t%bgn&p1&`vQ-p0w)_wPQ^6?1zR995>VzsouLEHZ0S*E2A$MfD?gxH zQ4wR;bBi*rgEkPg@7`6`kY{nc@94WrX<^+c;dpC55G8Ei;LahM5lMJdEil;;`or>J z-|iv2asUq zS4f_MBStd;+7>^}zC^!K4JdWN^#lhGsL%$nW%5K$1D^}oaYKTvV6HcaGu{J;!mzHz$^sZI8TlA(tHKR%V(k# z`T9FGxgQ%Cs&4Q6OJwexKG$j+U|->tlw06us;v+K@F<<5a6kYcjDQ+DDGq&$Djw_I z&QSZFu6wms=P830wPw!Hi5XaO^#~phm zK*yrdu24J6n0;;>?N2G9%cZ_jnf~RGSxB?EgDAHVYb{`()xE&rtqgFk{N*CXgQyRhVEmI!(U4Iq{1&?1W6zy{Atm7b-8N$~Ye zM$dslq3>eIjx=iS{j#qLNdd2R(JkSRW>6t|%wdKHhV zpN-PpEP5AbZjj2JyPW&!hMqw)vr8zASuV}{M+KZ8BNV~9h3UpMgA*gd0Q!ri2~uF< z9x>koaQ(RLoqGw&-{f{&4i@S1%bM(0jSl+wM~~;GtXZ>J=EW_DG#(JA2whmZ`}xSx z1UUT+lBhz_>qaHKUUgXnNIs~eD7ThucpNf~6sv#0N6xH$5V2Cao zJ9A5rEh7kk2nNd;@?lv8N%R(NygCk8s%?M!PA;gO z<^I>@cKfj>7jMKh{VI^=h5mraC9|qeS=LPcFmhaTKTqul95Qw03N^5-umq%q z2YnAVq0@+dD!&wMIvLf>$h6d+stpTlYwFBD(M`NvJbv|q$Zn4wW(dBgb36Jxget2(4~h1{r3A)9ZW-Ul>vS0K|Ck~gcJX$O z^YCnw=0|SfC;)#TNU&pBI^3}((4Fy_;JsC+r8VO^GmDXU&CaqlP*nHflRjnbYiiGb zWj!7~@j6AKH@DlOa@lpk>(?pJeP6+4O<2MXb{NYW^KnTIyiLIn5b%qTEz35>NZHW` z^I7&VrbXez8MU@UrS*sVMO<(|4VhDu@{!MtFXG0!u*v#~aoDR?r_H=#9S> zQ%iPU7p;jw`9PyobGNoO#+P0doq{ax>$C5(Zx$|TJy22tM%PXjQsw`IV9Z% zge|^Xq_Pi?A9X_sGiGTv0g{G!YBJukf&m|pl{4@j(ERg(;nic*d;0yVW5*A3Wx2ut zewl22%dgMk5XKmFKwtw$+HIkL^z$Po0E*rkjto-VFCEc-VAW)fJ_5>2vHm;WDA}@( zO_Tf^-XQf|WYL`?MF;;Rwv8yKVab(Px~*OoX*WLPl^|Cn0o>p{^bb!3TZi)`QY(no zWK(l=qzo$DMg8GU6HvKVR&sprg)s+(x|bIc4|H(vj#x=Yo*f`I~sDDya%o-jNTX*;PLJlx8PoIk#$>m!NU`~2qw}9%Mh3$ zBFI!{NcB`sHTMIF?hFXA0vJuUYU|XOxAdvn-g~Vr?PJ3gx=WshnA3jAmmSJeJHH6M z{0?gd15O>(*5HjTUC4RI0mRl!I-oIdTeSZl9tDet3(V8!p%a2WCid~R$;PNd%c+CP z1b*YM8GFhis>R0(#qj0#TzHX>e7EWvR^0Kevy2ncw?qsA*E$&kcW{(WJmu?4?O*my z<~=p48BaWddpp!WTFHHAzVYtSzbfDfi=fXl?vv5VG^rm9y_w*L3M0ZG6hTE#;R@%E-g zUlCVy2#l6HoN_RT@Bmy1)Kg^hfUn*d>zru0ZwkeBgl9&X58c%DYsER_9SYTXrLm*y zywe-A%x#15*KTmL$E*}`XuM=aKWa-CN*=9Cr;RioPD!&+sS0*8@NL&VG!hY_x_?~8 z?ZTZ$l1KA1$w~!`O}6|5_*6W^0slftPW@GKZD5NI{ivF6g1a<^D1jU0sO#hvg z+){Ss)?;dj)Sy^#{-Jo)#uyL#jtqh4Ha|M8x<}ONUGJRRe!#0!hC+7E(j@ zqr|?#WnR9#*v!yKu?TDpGeT%i&IwCeE2MWjI^DI<=iili*uFLsQwcgChtycXmh$3? zGQjOFLkC`=mBGF4u!DfK6F?BtZ8{_U31n=FK2r78_s+YoUl*NwTu8|qKaf7`rjD-H z`|cnz^c(QqfpmP7WsL!#68$&DD zcG*2O>LH*bf8#1mZm3P___a^3;4^&&jLJ@z3O&*i`gB*2f1BM1vb51f5P$@PGkAYg zDzX`XGn>?$^>TWJ$5`V(JQCo1)$X1XZyDmfbsn}FIOzuklync7t0@J?%?4yffzJ$B zf^g};?S}-Z5AV0TqxHofV-3Y{ z@>99cE>}w340}t$rr}*(!-t_=nM?rwgElA8BEaZ5%W1~!`i9)iRgckg|kcsgA$rAaGN`NL;v7vpEx>dW5YNuP4|8lfYxY_>3vv z+Bk3x8|#}O-oL=!2^}iK^moDm&%@M?0-7)QP*AZ(pQ6Y2@4vO=g4ic$9fsY3e5L!rBA validated rich text -> overlapping chunks -> page/span locat skip: available ? false : 'Build the parser with zig build to run the real CLI integration', }, async () => { const document = await extractPdfDocument(fixturePdf(), { binaryPath }); - assert.equal(document.manifest.schema_version, '0.17.0'); + assert.equal(document.manifest.schema_version, '0.19.0'); assert.equal(document.spans.length, 4); assert.equal(document.blocks.length, 4); assert.equal(document.sourceText, Array(4).fill('Repeated evidence.').join('\n\n')); @@ -141,7 +141,7 @@ test('actual multi-page stream preserves global ownership with page-local locato const receipt = await extractPdfPages(fixturePdf(), { binaryPath, onPage: async page => { pages.push(page); } }); assert.equal(receipt.complete, true); assert.equal(receipt.pageCount, 2); - assert.equal(receipt.schemaVersion, '0.17.0'); + assert.equal(receipt.schemaVersion, '0.19.0'); assert.deepEqual(pages.map(page => page.pageIndex), [0, 1]); assert.equal(new Set(pages.flatMap(page => page.spans.map(span => span.span_id))).size, 4); for (const page of pages) { diff --git a/integrations/javascript/stream.mjs b/integrations/javascript/stream.mjs index bce3f2d..c27440e 100644 --- a/integrations/javascript/stream.mjs +++ b/integrations/javascript/stream.mjs @@ -78,7 +78,7 @@ async function consume(binaryPath, args, options, inputSha256) { const firstRecord = !manifest; if (firstRecord) { if (type !== 'document_manifest' || record.streaming !== true || record.sequence_scope !== 'document' || - !/^0\.(14|15|16|17)\.\d+$/.test(record.schema_version ?? '') || !integer(record.page_count) || + !/^0\.(14|15|16|17|18|19)\.\d+$/.test(record.schema_version ?? '') || !integer(record.page_count) || typeof record.document_id !== 'string' || !record.document_id || record.input_sha256 !== inputSha256 || record.source_id !== inputSha256) fail('Invalid stream manifest'); if (record.page_count > options.maxPages) throw new PdfExtractionError('output_limit', 'Stream exceeded its page-count budget'); if (!Array.isArray(record.errors) || record.errors.length) fail('Stream manifest reports document errors'); diff --git a/integrations/javascript/stream.test.mjs b/integrations/javascript/stream.test.mjs index e57f136..44ebbd0 100644 --- a/integrations/javascript/stream.test.mjs +++ b/integrations/javascript/stream.test.mjs @@ -6,6 +6,7 @@ import { join } from 'node:path'; import { tmpdir } from 'node:os'; import { setTimeout as delay } from 'node:timers/promises'; import { extractPdfPages } from './stream.mjs'; +import { getPageGeometry } from './coordinates.mjs'; const input = Buffer.from('%PDF-stream-test'); const hash = (value) => createHash('sha256').update(value).digest('hex'); @@ -55,6 +56,50 @@ async function run(t, records = fixture(), extra = {}, code = '') { return { ...driver, pages, receipt }; } +function coordinateFixture() { + const records = fixture(); + for (const record of records) { + record.schema_version = '0.18.0'; + if (record.record_type === 'span') Object.assign(record, { + reconciled_block_id: `block-${record.page_index}`, + paint: { rendering_mode: 0, paint_state: 'painting', clip_state: 'unclipped', clip_bbox: null }, + content_scope: { page_index: record.page_index, stream_object_num: null, stream_generation: 0, invocation_index: 0 }, + }); + if (record.record_type === 'page_finished') record.artifact_counts.route_traces = 1; + if (record.record_type === 'document_finished') record.artifact_counts.route_traces = 2; + } + for (let p = 0; p < 2; p++) { + const end = records.find(record => record.record_type === 'page_finished' && record.page_index === p); + const route = { ...end, record_type: 'route_trace', schema_name: 'route_trace', event_type: 'route_trace', + provenance: { ...end.provenance, artifact_id: `route-page-${p}` }, trace_kind: 'page_route', + coverage: { status: 'text_available', completeness: 'unverified', native_span_count: 1, recognized_span_count: 0, + requested_recovery: false, recovery_state: 'not_required', unresolved_region_count: 0, + non_painting_span_count: 0, clipped_out_span_count: 0, uncertain_paint_span_count: 0 }, + page_geometry: { coordinate_space: 'pdf_user_space', user_unit: 1, rotation_degrees: p === 0 ? 90 : 270, + media_box: { x0: 0, y0: 0, x1: 20, y1: 40 }, crop_box: { x0: 0, y0: 0, x1: 20, y1: 40 }, + display: { box: 'crop_box', origin: 'top_left', unit: 'point', width: 40, height: 20, + pdf_to_display: p === 0 ? [0, 1, 1, 0, 0, 0] : [0, -1, -1, 0, 40, 20], + display_to_pdf: p === 0 ? [0, 1, 1, 0, 0, 0] : [0, -1, -1, 0, 20, 40] } } }; + records.splice(records.indexOf(end), 0, route); + } + records.forEach((record, index) => { record.event_index = index; }); + return records; +} + +test('0.18 stream pages retain their explicit global page geometry', async t => { + const result = await run(t, coordinateFixture()); + assert.equal(result.receipt.schemaVersion, '0.18.0'); + assert.deepEqual(result.pages.map(page => getPageGeometry(page, page.pageIndex).rotation_degrees), [90, 270]); +}); + +test('0.18 streaming refuses forged geometry before delivering that page', async t => { + const records = coordinateFixture(); + records.find(record => record.trace_kind === 'page_route' && record.page_index === 1).page_geometry.display.pdf_to_display[4] += 1; + const pages = []; + await assert.rejects(run(t, records, { onPage: page => pages.push(page.pageIndex) }), { code: 'protocol' }); + assert.deepEqual(pages, [0]); +}); + test('streams isolated page projections with global IDs and accepts only final receipt', async (t) => { const result = await run(t); assert.equal(result.receipt.complete, true); diff --git a/integrations/javascript/vector-contract.test.mjs b/integrations/javascript/vector-contract.test.mjs new file mode 100644 index 0000000..2e1088a --- /dev/null +++ b/integrations/javascript/vector-contract.test.mjs @@ -0,0 +1,259 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createHash } from 'node:crypto'; +import { mkdtemp, writeFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { parseArtifactJsonl, locateSourceRange } from './extract.mjs'; +import { extractPdfPages } from './stream.mjs'; + +const input = Buffer.from('%PDF-vector-contract'); +const sha = value => createHash('sha256').update(value).digest('hex'); +const inputSha256 = sha(input); +const bbox = { x0: 10, y0: 20, x1: 70, y1: 40 }; +const serialize = records => records.map(record => JSON.stringify(record)).join('\n') + '\n'; +const parse = records => parseArtifactJsonl(serialize(records), { inputSha256 }); +const countKeys = ['spans', 'blocks', 'rag_chunks', 'tables', 'form_fields', 'route_traces', 'specialist_requests', 'specialist_attempts', 'specialist_responses', 'specialist_results', 'debug_assets']; +function fixture(version = '0.19.0') { + const common = (type, id, refs = {}) => ({ schema_name: type, schema_version: version, record_type: type, + document_id: 'doc', source_id: inputSha256, provenance: { document_id: 'doc', source_id: inputSha256, + input_sha256: inputSha256, artifact_id: id, span_ids: [], block_ids: [], chunk_ids: [], ...refs } }); + const text = 'Native 😀 evidence'; + const refs = { span_ids: ['span-0'], block_ids: ['block-0'] }; + const view = (id, kind) => ({ text_view_id: `${id}:text`, kind, offset_unit: 'utf8_byte', range_convention: 'half_open', normalization: 'none', text_sha256: sha(text) }); + const coverage = { status: 'recovery_required', completeness: 'unverified', native_span_count: 1, recognized_span_count: 0, + requested_recovery: true, recovery_state: 'not_invoked', unresolved_region_count: 1, + non_painting_span_count: 0, clipped_out_span_count: 0, uncertain_paint_span_count: 0 }; + const artifact_counts = Object.fromEntries(countKeys.map(key => [key, 0])); + Object.assign(artifact_counts, { spans: 1, blocks: 1, rag_chunks: 1, route_traces: 2, specialist_requests: 1 }); + return [ + { ...common('document_manifest', 'document_manifest'), input_sha256: inputSha256, page_count: 1, artifact_counts, + text_contract: { offset_unit: 'utf8_byte', range_convention: 'half_open', page_index_base: 0, + geometry: { coordinate_space: 'pdf_user_space', unit: 'point', highlight_precision: 'span_bbox', display_transform: null } } }, + { ...common('span', 'span-0', refs), span_id: 'span-0', reconciled_block_id: 'block-0', page_index: 0, bbox, text, + text_view: view('span-0', 'reconciled_span'), paint: { rendering_mode: 0, paint_state: 'painting', clip_state: 'unclipped', clip_bbox: null }, + content_scope: { page_index: 0, stream_object_num: null, stream_generation: 0, invocation_index: 0 } }, + { ...common('block', 'block-0', refs), block_id: 'block-0', page_index: 0, bbox, text, text_view: view('block-0', 'reconciled_block'), + source_map: [{ text_start_byte: 0, text_end_byte: Buffer.byteLength(text), source_span_id: 'span-0', source_start_byte: 0, source_end_byte: Buffer.byteLength(text) }] }, + { ...common('rag_chunk', 'chunk-0', { ...refs, chunk_ids: ['chunk-0'] }), chunk_id: 'chunk-0', page_start: 0, page_end: 0, content: text, + source_span_ids: ['span-0'], source_block_ids: ['block-0'], text_view: view('chunk-0', 'rag_chunk'), + source_map: [{ text_start_byte: 0, text_end_byte: Buffer.byteLength(text), source_block_id: 'block-0', source_start_byte: 0, source_end_byte: Buffer.byteLength(text) }] }, + { ...common('route_trace', 'route-page-0', { route_reasons: ['unexplained_vector'] }), trace_kind: 'page_route', page_index: 0, region_index: null, + reasons: ['unexplained_vector'], page_geometry: null, coverage: { ...coverage }, vector_evidence: { + status: 'complete', shape_count: 6, unknown_count: 0, ignored_rule_count: 1, clipped_out_count: 0, native_accounted_count: 1, + simple_graphic_count: 0, ambiguous_shape_count: 1, candidate_region_count: 1, + max_shapes: 512, max_segments_per_path: 512, max_operators: 100000, max_candidate_regions: 64, + operator_count: 40, content_bytes: 600, max_content_bytes: 8388608, max_graphics_depth: 32, max_form_invocations: 256 } }, + { ...common('route_trace', 'route-region-0', { route_reasons: ['unexplained_vector'] }), trace_kind: 'region_route', page_index: 0, region_index: 0, + bbox, route: 'queue_ocr', reasons: ['unexplained_vector'], coverage: { ...coverage, native_span_count: 0 }, + specialist_status: 'requested', specialist_request_ids: ['ocr-request-0'], + vector_evidence: { bbox: { ...bbox }, shape_count: 3, segment_count: 12, subpath_count: 3, native_coverage: 0, decision: 'unexplained_vector' } }, + { ...common('specialist_request', 'ocr-request-0'), request_id: 'ocr-request-0', page_index: 0, region_index: 0, requested_kind: 'ocr', crop_geometry: null }, + ]; +} + +test('0.19 preserves vector metadata and exact native source text and locators', () => { + const records = fixture(); + const document = parse(records); + assert.deepEqual(document.records[4].vector_evidence, records[4].vector_evidence); + assert.deepEqual(document.records[5].vector_evidence, records[5].vector_evidence); + assert.equal(document.sourceText, records[1].text); + assert.equal(document.chunks[0].content, records[1].text); + const located = locateSourceRange(document, 0, document.sourceText.length, { sourceTextSha256: document.sourceTextSha256 }); + assert.equal(located.segments[0].sourceSpanId, 'span-0'); + assert.deepEqual(located.segments[0].bbox, bbox); + assert.equal(document.pageCoverage, 'unverified'); +}); + +test('0.19 accepts unknown, partial and limited evidence without inventing candidates', () => { + for (const status of ['complete', 'partial', 'limited']) { + const records = fixture(); + records[4].vector_evidence.status = status; + records[4].vector_evidence.unknown_count = status === 'complete' ? 0 : 1000; + assert.ok(parse(records)); + } + const records = fixture(); + records[4].vector_evidence = records[5].vector_evidence = null; + assert.equal(parse(records).records[4].vector_evidence, null); +}); + +test('0.19 accepts a retained subset of a page candidate collection', () => { + const records = fixture(); + records[4].vector_evidence.candidate_region_count = 2; + records[4].vector_evidence.shape_count = 9; + assert.ok(parse(records)); +}); + +test('0.19 accepts reached resource caps without exceeding them', () => { + const records = fixture(); + Object.assign(records[4].vector_evidence, { status: 'limited', operator_count: 100000, content_bytes: 8388608 }); + assert.ok(parse(records)); + Object.assign(records[4].vector_evidence, { operator_count: 0, content_bytes: 0 }); + assert.ok(parse(records)); +}); + +test('0.19 retains a smaller vector extent inside a reused image OCR route', () => { + const records = fixture(); + records[5].bbox = { x0: 0, y0: 0, x1: 100, y1: 100 }; + const document = parse(records); + assert.deepEqual(document.records[5].vector_evidence.bbox, bbox); + assert.deepEqual(document.records[5].bbox, records[5].bbox); +}); + +test('0.19 tolerates two-decimal route rounding and large nonzero origins', () => { + for (const origin of [0, -10000, 1000000]) { + const records = fixture(); + records[5].bbox = { x0: origin + 10, y0: origin + 20, x1: origin + 70, y1: origin + 40 }; + records[5].vector_evidence.bbox = { x0: origin + 9.995, y0: origin + 19.995, x1: origin + 70.005, y1: origin + 40.005 }; + assert.ok(parse(records)); + } +}); + +test('0.19 accepts a page summary when another OCR route covers its candidates', () => { + const records = fixture(); + records[5].vector_evidence = null; + records[5].region_index = null; + records[6].region_index = null; + assert.ok(parse(records)); +}); + +test('0.19 accepts an empty known summary and requires no evidence on other stages', () => { + const records = fixture(); + for (const key of ['shape_count', 'candidate_region_count', 'ignored_rule_count', 'native_accounted_count', 'ambiguous_shape_count']) records[4].vector_evidence[key] = 0; + records[5].vector_evidence = null; + const stage = { ...records[5], trace_kind: 'layout', provenance: { ...records[5].provenance, artifact_id: 'route-stage-0' } }; + delete stage.vector_evidence; + records.push(stage); + records[0].artifact_counts.route_traces++; + assert.ok(parse(records)); +}); + +for (const minor of [14, 15, 16, 17, 18]) test(`reader retains 0.${minor} compatibility without vector fields`, () => { + const records = fixture(`0.${minor}.0`); + delete records[4].vector_evidence; + delete records[5].vector_evidence; + assert.equal(parse(records).sourceText, records[1].text); +}); + +for (const [name, mutate] of [ + ['missing page evidence', r => { delete r[4].vector_evidence; }], + ['missing region evidence', r => { delete r[5].vector_evidence; }], + ['unknown status', r => { r[4].vector_evidence.status = 'visible'; }], + ['negative count', r => { r[4].vector_evidence.unknown_count = -1; }], + ['fractional count', r => { r[4].vector_evidence.shape_count = 3.5; }], + ['unsafe count', r => { r[4].vector_evidence.unknown_count = 2 ** 53; }], + ['missing summary count', r => { delete r[4].vector_evidence.simple_graphic_count; }], + ['changed shape cap', r => { r[4].vector_evidence.max_shapes = 1024; }], + ['changed segment cap', r => { r[4].vector_evidence.max_segments_per_path = 1024; }], + ['changed operator cap', r => { r[4].vector_evidence.max_operators = 100001; }], + ['changed region cap', r => { r[4].vector_evidence.max_candidate_regions = 65; }], + ['missing operator count', r => { delete r[4].vector_evidence.operator_count; }], + ['negative operator count', r => { r[4].vector_evidence.operator_count = -1; }], + ['fractional operator count', r => { r[4].vector_evidence.operator_count = 0.5; }], + ['operator budget overflow', r => { r[4].vector_evidence.operator_count = 100001; }], + ['missing content byte count', r => { delete r[4].vector_evidence.content_bytes; }], + ['negative content byte count', r => { r[4].vector_evidence.content_bytes = -1; }], + ['fractional content byte count', r => { r[4].vector_evidence.content_bytes = 0.5; }], + ['content byte budget overflow', r => { r[4].vector_evidence.content_bytes = 8388609; }], + ['missing content byte cap', r => { delete r[4].vector_evidence.max_content_bytes; }], + ['changed content byte cap', r => { r[4].vector_evidence.max_content_bytes = 8388609; }], + ['missing graphics depth cap', r => { delete r[4].vector_evidence.max_graphics_depth; }], + ['changed graphics depth cap', r => { r[4].vector_evidence.max_graphics_depth = 33; }], + ['missing Form invocation cap', r => { delete r[4].vector_evidence.max_form_invocations; }], + ['changed Form invocation cap', r => { r[4].vector_evidence.max_form_invocations = 257; }], + ['shape cap overflow', r => { r[4].vector_evidence.shape_count = 513; }], + ['region cap overflow', r => { r[4].vector_evidence.candidate_region_count = 65; }], + ['classified count overflow', r => { r[4].vector_evidence.ambiguous_shape_count = 6; }], + ['region count exceeds shapes', r => { r[4].vector_evidence.candidate_region_count = 7; }], + ['missing page candidate reason', r => { r[4].reasons = []; }], + ['missing page provenance reason', r => { r[4].provenance.route_reasons = []; }], + ['null parent summary for known region', r => { r[4].vector_evidence = null; }], + ['missing parent route', r => { r.splice(4, 1); r[0].artifact_counts.route_traces--; }], + ['empty candidate shapes', r => { r[5].vector_evidence.shape_count = 0; }], + ['missing candidate bbox', r => { delete r[5].bbox; }], + ['empty candidate bbox', r => { r[5].bbox = { ...bbox, x1: bbox.x0 }; }], + ['missing exact vector bbox', r => { delete r[5].vector_evidence.bbox; }], + ['null exact vector bbox', r => { r[5].vector_evidence.bbox = null; }], + ['empty exact vector bbox', r => { r[5].vector_evidence.bbox.x1 = bbox.x0; }], + ['inverted exact vector bbox', r => { r[5].vector_evidence.bbox.y1 = bbox.y0 - 1; }], + ['nonfinite exact vector bbox', r => { r[5].vector_evidence.bbox.x0 = Infinity; }], + ['vector bbox beyond left route edge', r => { r[5].vector_evidence.bbox.x0 = bbox.x0 - 0.006; }], + ['vector bbox beyond right route edge', r => { r[5].vector_evidence.bbox.x1 = bbox.x1 + 0.006; }], + ['vector bbox beyond bottom route edge', r => { r[5].vector_evidence.bbox.y0 = bbox.y0 - 0.006; }], + ['vector bbox beyond top route edge', r => { r[5].vector_evidence.bbox.y1 = bbox.y1 + 0.006; }], + ['missing candidate count', r => { delete r[5].vector_evidence.segment_count; }], + ['fractional candidate count', r => { r[5].vector_evidence.segment_count = 12.5; }], + ['segment cap overflow', r => { r[5].vector_evidence.segment_count = 1537; }], + ['subpaths exceed segments', r => { r[5].vector_evidence.subpath_count = 13; }], + ['candidate shapes exceed remaining page shapes', r => { r[5].vector_evidence.shape_count = 4; }], + ['negative coverage', r => { r[5].vector_evidence.native_coverage = -0.1; }], + ['excessive coverage', r => { r[5].vector_evidence.native_coverage = 1.1; }], + ['nonfinite coverage', r => { r[5].vector_evidence.native_coverage = Infinity; }], + ['unknown decision', r => { r[5].vector_evidence.decision = 'recognized_text'; }], + ['missing regional reason', r => { r[5].reasons = []; }], + ['missing regional provenance reason', r => { r[5].provenance.route_reasons = []; }], + ['native-only candidate', r => { r[5].route = 'native'; }], + ['candidate without request status', r => { r[5].specialist_status = 'none'; }], + ['unlinked candidate request', r => { r[5].specialist_request_ids = []; }], + ['request wrong region', r => { r[6].region_index = 1; }], + ['request wrong specialist', r => { r[6].requested_kind = 'table'; }], + ['zero candidate summary', r => { r[4].vector_evidence.candidate_region_count = 0; }], + ['missing inherited page geometry', r => { delete r[4].page_geometry; }], + ['missing inherited request geometry', r => { delete r[6].crop_geometry; }], + ['missing inherited paint', r => { delete r[1].paint; }], + ['missing inherited scope', r => { delete r[1].content_scope; }], + ['malformed inherited paint', r => { r[1].paint.rendering_mode = 3; }], + ['malformed inherited scope', r => { r[1].content_scope.invocation_index = 1; }], + ['missing inherited coverage count', r => { delete r[4].coverage.uncertain_paint_span_count; }], +]) test(`0.19 rejects ${name}`, () => { + const records = fixture(); + mutate(records); + assert.throws(() => parse(records), { code: 'protocol' }); +}); + +test('0.19 rejects duplicate candidate region identities even with unique route IDs', () => { + const records = fixture(); + const duplicate = structuredClone(records[5]); + duplicate.provenance.artifact_id = 'route-region-1'; + records.push(duplicate); + records[0].artifact_counts.route_traces++; + records[4].vector_evidence.candidate_region_count = 2; + records[4].vector_evidence.shape_count = 9; + assert.throws(() => parse(records), { code: 'protocol' }); +}); + +async function streamFixture(t, records) { + const directory = await mkdtemp(join(tmpdir(), 'pdf-vector-contract-')); + t.after(() => rm(directory, { recursive: true, force: true })); + const counts = records[0].artifact_counts; + const route_counts = { native_pages: 0, page_routes: 1, region_routes: 1, ocr_regions: 2, table_regions: 0, formula_regions: 0 }; + const lifecycle = (type, id, fields) => ({ ...records[0], schema_name: type, record_type: type, + provenance: { ...records[0].provenance, artifact_id: id }, ...fields }); + const events = [{ ...records[0], streaming: true, errors: [] }, + lifecycle('page_started', 'page-started-0', { page_index: 0, status: 'running' }), ...records.slice(1), + lifecycle('page_finished', 'page-finished-0', { page_index: 0, status: 'completed', artifact_counts: counts, route_counts }), + lifecycle('document_finished', 'document_finished', { status: 'completed', errors: [], artifact_counts: counts, route_counts })]; + events.forEach((record, index) => Object.assign(record, { event_index: index, event_type: record.record_type, + sequence_scope: record.record_type.startsWith('document_') ? 'document' : 'page' })); + const binaryPath = join(directory, 'parser'); + await writeFile(binaryPath, `#!${process.execPath}\nprocess.stdout.write(${JSON.stringify(serialize(events))});\n`, { mode: 0o700 }); + return binaryPath; +} + +test('0.19 streaming validates vector evidence before delivering a page', async t => { + const records = fixture(); + const pages = []; + const receipt = await extractPdfPages(input, { binaryPath: await streamFixture(t, records), onPage: page => pages.push(page) }); + assert.equal(receipt.schemaVersion, '0.19.0'); + assert.equal(receipt.complete, true); + assert.deepEqual(pages[0].records.find(record => record.trace_kind === 'region_route').vector_evidence, records[5].vector_evidence); + assert.equal(pages[0].sourceText, records[1].text); +}); + +test('0.19 streaming refuses a malformed vector page before callback', async t => { + const records = fixture(); + records[5].vector_evidence.native_coverage = 2; + const pages = []; + await assert.rejects(extractPdfPages(input, { binaryPath: await streamFixture(t, records), onPage: page => pages.push(page) }), { code: 'protocol' }); + assert.equal(pages.length, 0); +}); diff --git a/integrations/javascript/vector-routing.test.mjs b/integrations/javascript/vector-routing.test.mjs new file mode 100644 index 0000000..e63e5e6 --- /dev/null +++ b/integrations/javascript/vector-routing.test.mjs @@ -0,0 +1,321 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { access, readFile, mkdtemp, writeFile, rm } from 'node:fs/promises'; +import { constants } from 'node:fs'; +import { execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { extractPdfDocument, extractPdfPages, locateSourceRange, parseArtifactJsonl, getPageGeometry, mapPdfBBoxToDisplay, mapDisplayBBoxToPdf } from './index.mjs'; + +const binaryPath = fileURLToPath(new URL('../../zig-out/bin/pdf-parser', import.meta.url)); +const fixtureRoot = new URL('../../benchmark/eval/vector_routing/', import.meta.url); +const manifest = JSON.parse(await readFile(new URL('manifest.json', fixtureRoot), 'utf8')); +let available = true; +try { await access(binaryPath, constants.X_OK); } catch { available = false; } +const toolAvailable = name => { + try { execFileSync(name, [name === 'pdftoppm' ? '-v' : '--version'], { stdio: 'ignore', timeout: 5000 }); return true; } catch { return false; } +}; +const rasterAvailable = toolAvailable('pdftoppm'); +const ocrAvailable = toolAvailable('tesseract'); +const fixture = name => readFile(new URL(`${name}.pdf`, fixtureRoot)); +const fixtureInfo = name => manifest.fixtures.find(item => item.id === name); +const vectorRoutes = records => records.filter(record => record.record_type === 'route_trace' && record.trace_kind === 'region_route' && record.reasons.includes('unexplained_vector')); +const pageRoutes = records => records.filter(record => record.record_type === 'route_trace' && record.trace_kind === 'page_route'); +const ocrAttempts = records => records.filter(record => record.record_type === 'specialist_attempt' && record.specialist_kind === 'ocr'); + +function assertNative(document, name) { + for (const expected of fixtureInfo(name).native_text) assert.ok(document.sourceText.includes(expected), `${name}: missing ${expected}`); +} + +function assertLocated(document, phrase) { + const start = document.sourceText.indexOf(phrase); + assert.ok(start >= 0, document.sourceText); + assert.ok(document.chunks.some(chunk => chunk.content.includes(phrase)), `missing chunk: ${phrase}`); + const locator = locateSourceRange(document, start, start + phrase.length, { sourceTextSha256: document.sourceTextSha256 }); + assert.ok(locator.segments.some(segment => segment.sourceSpanId && segment.bbox)); + for (const segment of locator.segments.filter(segment => segment.sourceSpanId)) { + const span = document.spans.find(item => item.span_id === segment.sourceSpanId); + assert.equal(document.sourceText.slice(segment.start, segment.end), span.text.slice(segment.sourceStart, segment.sourceEnd)); + assert.equal(segment.pageIndex, span.page_index); + } +} + +function assertVectorEvidence(route) { + const evidence = route.vector_evidence; + assert.ok(evidence, route.route_trace_id); + assert.equal(evidence.decision, 'unexplained_vector'); + for (const key of ['shape_count', 'segment_count', 'subpath_count']) assert.ok(Number.isInteger(evidence[key]) && evidence[key] > 0, key); + assert.ok(evidence.native_coverage >= 0 && evidence.native_coverage <= 1); + assert.ok(Number.isInteger(route.region_index)); + assert.ok(route.bbox.x0 < route.bbox.x1 && route.bbox.y0 < route.bbox.y1); +} + +async function cliRecords(t, name, flags = [], format = 'artifact-jsonl') { + const directory = await mkdtemp(join(tmpdir(), 'pdf-parser-vector-cli-')); + t.after(() => rm(directory, { recursive: true, force: true })); + const input = join(directory, 'input.pdf'); + const bytes = await fixture(name); + await writeFile(input, bytes); + const raw = execFileSync(binaryPath, ['extract-adaptive', '--input', input, '--format', format, ...flags], { + encoding: 'utf8', timeout: 45000, maxBuffer: 10 * 1024 * 1024, + }); + const records = raw.trim().split('\n').map(JSON.parse); + assert.equal(records[0].schema_version, '0.19.0'); + const document = format === 'artifact-jsonl' ? parseArtifactJsonl(raw, { inputSha256: createHash('sha256').update(bytes).digest('hex') }) : null; + return { bytes, raw, records, document }; +} + +async function tools(t, mode = 'success', extraFlags = []) { + const directory = await mkdtemp(join(tmpdir(), 'pdf-parser-vector-engine-')); + t.after(() => rm(directory, { recursive: true, force: true })); + const engine = join(directory, 'engine.mjs'); + const log = join(directory, 'calls.txt'); + await writeFile(engine, `#!/usr/bin/env node +import { readFileSync, appendFileSync } from 'node:fs'; +appendFileSync(${JSON.stringify(log)}, process.argv[2] + '\\n'); +${mode === 'failed' ? 'process.exit(7);' : ''} +const image = readFileSync(process.argv[2]); +const width = image.readUInt32BE(16), height = image.readUInt32BE(20); +if (width < 6 || height < 6) process.exit(8); +process.stdout.write('level\\tpage_num\\tblock_num\\tpar_num\\tline_num\\tword_num\\tleft\\ttop\\twidth\\theight\\tconf\\ttext\\n'); +process.stdout.write('5\\t1\\t1\\t1\\t1\\t1\\t2\\t2\\t' + ${mode === 'disjoint' ? '1' : 'Math.min(80,width-4)'} + '\\t' + ${mode === 'disjoint' ? '1' : 'Math.min(14,height-4)'} + '\\t99\\tVECTOR EVIDENCE\\n'); +`, { mode: 0o700 }); + const wrapper = join(directory, 'parser.mjs'); + const executable = mode === 'unavailable' ? join(directory, 'missing-engine') : engine; + const flags = ['--ocr-executable', executable, '--ocr-policy', 'single', '--ocr-dpi', '144', ...extraFlags]; + await writeFile(wrapper, `#!/usr/bin/env node +import { spawnSync } from 'node:child_process'; +const result = spawnSync(${JSON.stringify(binaryPath)}, [...process.argv.slice(2), ...${JSON.stringify(flags)}], { maxBuffer: 10*1024*1024, timeout: 45000 }); +if (result.error) { process.stderr.write(String(result.error)); process.exit(9); } +process.stdout.write(result.stdout); process.stderr.write(result.stderr); process.exitCode = result.status ?? 9; +`, { mode: 0o700 }); + return { binaryPath: wrapper, engine, log }; +} + +test('vector fixtures retain real curved glyph paths and reproducible bounded input identities', async () => { + assert.equal(manifest.font_program_embedded, false); + assert.match(await readFile(new URL(manifest.font_license_file, fixtureRoot), 'utf8'), /Copyright \(c\) 2003 by Bitstream/u); + assert.equal(manifest.fixtures.length, 23); + for (const item of manifest.fixtures) { + const bytes = await fixture(item.id); + assert.equal(bytes.length, item.bytes); + assert.ok(bytes.length <= 100000, item.id); + assert.equal(createHash('sha256').update(bytes).digest('hex'), item.sha256); + assert.ok(!bytes.toString('ascii').includes('/FontFile'), item.id); + } + const bytes = (await fixture('outlined-text')).toString('ascii'); + assert.match(bytes, / c\nh/u); + assert.ok(!bytes.includes('(VECTOR EVIDENCE)')); + assert.ok(!bytes.includes(' Tj')); +}); + +test('outlined text and mixed native paths emit bounded unresolved vector routes in both formats', { skip: !available }, async t => { + for (const name of ['outlined-text', 'mixed-native-outlines', 'partial-native-clip', 'chart-labels', 'transformed-form', 'quarter-turn-outlines', 'rule-through-outlines', 'stroked-letterforms', 'compound-fill']) { + for (const format of ['artifact-jsonl', 'stream-jsonl']) { + const { records, document } = await cliRecords(t, name, ['--no-ocr'], format); + const routes = vectorRoutes(records); + assert.ok(routes.length > 0, `${name}/${format}: no vector route`); + routes.forEach(assertVectorEvidence); + const page = pageRoutes(records)[0]; + assert.ok(page.vector_evidence.shape_count > 0); + assert.ok(page.vector_evidence.candidate_region_count > 0); + assert.equal(page.vector_evidence.max_shapes, 512); + assert.equal(page.vector_evidence.max_segments_per_path, 512); + assert.equal(page.vector_evidence.max_operators, 100000); + assert.equal(page.coverage.requested_recovery, true); + assert.equal(page.coverage.completeness, 'unverified'); + if (document) assertNative(document, name); + } + } +}); + +test('rules highlights boxes borders logos and clipped-out shapes do not invent vector OCR routes', { skip: !available }, async t => { + for (const name of ['thin-stroked-rule', 'thin-filled-rule', 'native-highlight', 'filled-box', 'border', 'single-logo', 'clipped-out']) { + const { records, document } = await cliRecords(t, name, ['--no-ocr']); + assert.equal(vectorRoutes(records).length, 0, name); + assertNative(document, name); + assert.ok(pageRoutes(records)[0].vector_evidence, name); + } +}); + +test('dense graphics and fused panels expose conservative vector evidence without a completeness claim', { skip: !available }, async t => { + for (const name of ['dense-graphics', 'fused-panel']) { + const { records, document } = await cliRecords(t, name, ['--no-ocr']); + const evidence = pageRoutes(records)[0].vector_evidence; + assert.ok(evidence.shape_count > 0); + assert.ok(evidence.ambiguous_shape_count > 0 || vectorRoutes(records).length > 0 || evidence.simple_graphic_count > 0); + assert.equal(pageRoutes(records)[0].coverage.completeness, 'unverified'); + assertNative(document, name); + } +}); + +test('non-axis-aligned Form clipping explicitly abstains from vector OCR', { skip: !available }, async t => { + const { records, document } = await cliRecords(t, 'sheared-form', ['--no-ocr']); + const evidence = pageRoutes(records)[0].vector_evidence; + assert.equal(evidence.status, 'partial'); + assert.ok(evidence.unknown_count > 0); + assert.ok(evidence.ambiguous_shape_count > 0); + assert.equal(vectorRoutes(records).length, 0); + assertNative(document, 'sheared-form'); + assert.equal(pageRoutes(records)[0].coverage.completeness, 'unverified'); +}); + +test('native vector collection caps report limited evidence while preserving native text', { skip: !available }, async t => { + for (const name of ['shape-cap', 'segment-cap']) { + const { records, document } = await cliRecords(t, name, ['--no-ocr']); + const evidence = pageRoutes(records)[0].vector_evidence; + assert.equal(evidence.status, 'limited', name); + assert.ok(evidence.shape_count <= evidence.max_shapes); + assert.equal(pageRoutes(records)[0].coverage.completeness, 'unverified'); + assertNative(document, name); + } +}); + +test('deterministic regional vector OCR binds routes attempts source text chunks and locators', { skip: !available || !rasterAvailable }, async t => { + const fake = await tools(t); + const bytes = await fixture('mixed-native-outlines'); + const document = await extractPdfDocument(bytes, { binaryPath: fake.binaryPath, enableOcr: true }); + assertNative(document, 'mixed-native-outlines'); + assertLocated(document, 'VECTOR EVIDENCE'); + const routes = vectorRoutes(document.records); + assert.ok(routes.length > 0); + const attempts = ocrAttempts(document.records).filter(attempt => attempt.selected); + assert.ok(attempts.length > 0); + for (const attempt of attempts) { + const route = routes.find(item => item.region_index === attempt.region_index); + assert.ok(route); + assert.ok(route.specialist_request_ids.includes(attempt.request_id)); + assert.equal(attempt.page_index, route.page_index); + assert.ok(attempt.crop.image_byte_length > 24); + assert.match(attempt.crop.image_sha256, /^[a-f0-9]{64}$/u); + } + const pages = []; + const receipt = await extractPdfPages(bytes, { binaryPath: fake.binaryPath, enableOcr: true, onPage: page => pages.push(page) }); + assert.equal(receipt.complete, true); + assert.equal(receipt.schemaVersion, '0.19.0'); + assertNative(pages[0], 'mixed-native-outlines'); + assertLocated(pages[0], 'VECTOR EVIDENCE'); + const cropPaths = (await readFile(fake.log, 'utf8')).trim().split('\n'); + assert.ok(cropPaths.length >= 2); + for (const path of cropPaths) await assert.rejects(access(path), { code: 'ENOENT' }); +}); + +test('partially clipped native text cannot account for visible outlines painted after restore', { skip: !available || !rasterAvailable || !ocrAvailable }, async t => { + const bytes = await fixture('partial-native-clip'); + const document = await extractPdfDocument(bytes, { binaryPath, enableOcr: true }); + assertNative(document, 'partial-native-clip'); + assertLocated(document, 'VECTOR EVIDENCE'); + const routes = vectorRoutes(document.records); + assert.ok(routes.length > 0); + assert.ok(routes.every(route => route.vector_evidence.native_coverage < 0.01)); + assert.ok(document.spans.some(span => span.text === 'VECTOR' && span.chosen_source === 'fresh_ocr')); + const pages = []; + await extractPdfPages(bytes, { binaryPath, enableOcr: true, onPage: page => pages.push(page) }); + assertNative(pages[0], 'partial-native-clip'); + assertLocated(pages[0], 'VECTOR EVIDENCE'); + assert.ok(pages[0].spans.some(span => span.text === 'VECTOR' && span.chosen_source === 'fresh_ocr')); +}); + +test('failed unavailable and pixel-refused vector OCR preserve native evidence and unresolved routes', { skip: !available || !rasterAvailable }, async t => { + const bytes = await fixture('mixed-native-outlines'); + for (const mode of ['failed', 'unavailable', 'pixels']) { + const fake = await tools(t, mode); + const document = await extractPdfDocument(bytes, { binaryPath: fake.binaryPath, enableOcr: true, allowPartial: true, + ...(mode === 'pixels' ? { ocrLimits: { maxImagePixels: 1 } } : {}) }); + assertNative(document, 'mixed-native-outlines'); + assert.ok(!document.sourceText.includes('VECTOR EVIDENCE')); + assert.ok(vectorRoutes(document.records).length > 0); + assert.equal(document.pageOutcomes[0].completeness, 'unverified'); + assert.notEqual(document.pageOutcomes[0].status, 'recovered'); + const attempts = ocrAttempts(document.records); + assert.ok(attempts.length > 0); + assert.ok(attempts.every(attempt => !attempt.selected)); + if (mode === 'pixels') { + assert.ok(attempts.some(attempt => attempt.errors.some(error => error.code === 'rasterizer_pixel_limit'))); + await assert.rejects(access(fake.log), { code: 'ENOENT' }); + } + } +}); + +test('repeated vector crops have global identities and preserve every quota refusal', { skip: !available || !rasterAvailable }, async t => { + const fake = await tools(t); + const bytes = await fixture('repeated-multipage'); + for (const limits of [{ maxRegionsPerPage: 1 }, { maxAttemptsPerPage: 1 }]) { + const document = await extractPdfDocument(bytes, { binaryPath: fake.binaryPath, enableOcr: true, allowPartial: true, ocrLimits: limits }); + const routes = vectorRoutes(document.records); + assert.ok(routes.length >= 4); + assert.equal(new Set(routes.map(route => route.region_index)).size, routes.length); + const attempts = ocrAttempts(document.records); + assert.equal(new Set(attempts.map(attempt => attempt.attempt_id)).size, attempts.length); + for (const route of routes) assert.ok(attempts.some(attempt => attempt.region_index === route.region_index), `missing attempt for ${route.route_trace_id}`); + const expected = limits.maxRegionsPerPage ? 'ocr_region_limit' : 'ocr_attempt_limit'; + assert.ok(attempts.some(attempt => attempt.attempt_status === 'not_invoked' && [...attempt.errors, ...attempt.warnings].some(error => error.code === expected))); + assert.ok(document.pageOutcomes.every(outcome => outcome.completeness === 'unverified' && outcome.status !== 'recovered')); + assertNative(document, 'repeated-multipage'); + } + const pages = []; + await extractPdfPages(bytes, { binaryPath: fake.binaryPath, enableOcr: true, onPage: page => pages.push(page) }); + assert.equal(pages.length, 2); + const attempts = pages.flatMap(page => ocrAttempts(page.records)); + assert.equal(new Set(attempts.map(attempt => attempt.attempt_id)).size, attempts.length); + for (const page of pages) assertLocated(page, 'VECTOR EVIDENCE'); +}); + +test('overlapping bitmap and vector candidates share one regional OCR recovery', { skip: !available || !rasterAvailable }, async t => { + const fake = await tools(t); + const document = await extractPdfDocument(await fixture('image-overlap'), { binaryPath: fake.binaryPath, enableOcr: true }); + const selected = ocrAttempts(document.records).filter(attempt => attempt.selected); + assert.equal(selected.length, 1); + assert.equal(vectorRoutes(document.records).length, 1); + assertLocated(document, 'VECTOR EVIDENCE'); + assertNative(document, 'image-overlap'); +}); + +test('OCR text outside a vector candidate cannot recover its reused bitmap route', { skip: !available || !rasterAvailable }, async t => { + const fake = await tools(t, 'disjoint'); + const bytes = await fixture('image-overlap'); + const batch = await extractPdfDocument(bytes, { binaryPath: fake.binaryPath, enableOcr: true }); + const pages = []; + await extractPdfPages(bytes, { binaryPath: fake.binaryPath, enableOcr: true, onPage: page => pages.push(page) }); + for (const records of [batch.records, pages[0].records]) { + assert.equal(ocrAttempts(records).filter(attempt => attempt.selected).length, 1); + const routes = vectorRoutes(records); + assert.equal(routes.length, 1); + assert.equal(routes[0].coverage.status, 'recovery_required'); + assert.equal(routes[0].coverage.unresolved_region_count, 1); + assert.equal(routes[0].coverage.completeness, 'unverified'); + } + assert.equal(batch.pageOutcomes[0].status, 'recovery_required'); +}); + +test('actual Tesseract recovers path-only and mixed outlined text in both public SDK formats', { skip: !available || !rasterAvailable || !ocrAvailable }, async () => { + for (const name of ['outlined-text', 'mixed-native-outlines', 'transformed-form', 'quarter-turn-outlines', 'compound-fill']) { + const bytes = await fixture(name); + const document = await extractPdfDocument(bytes, { binaryPath, enableOcr: true }); + assertNative(document, name); + assertLocated(document, 'VECTOR EVIDENCE'); + assert.ok(vectorRoutes(document.records).length > 0); + assert.ok(ocrAttempts(document.records).some(attempt => attempt.selected && Number.isInteger(attempt.region_index))); + if (name === 'quarter-turn-outlines') { + const geometry = getPageGeometry(document, 0); + assert.equal(geometry.rotation_degrees, 90); + const word = document.spans.find(span => span.text === 'VECTOR' && span.chosen_source === 'fresh_ocr'); + assert.ok(word); + const expected = fixtureInfo(name).outlined_bboxes[0]; + assert.ok(word.bbox.x0 >= expected[0] - 3 && word.bbox.x1 <= expected[2] + 3); + assert.ok(word.bbox.y0 >= expected[1] - 3 && word.bbox.y1 <= expected[3] + 3); + assert.ok(word.bbox.y1 - word.bbox.y0 > word.bbox.x1 - word.bbox.x0); + const display = mapPdfBBoxToDisplay(word.bbox, geometry, { scale: 1 }); + assert.ok(display.x1 - display.x0 > display.y1 - display.y0); + const roundtrip = mapDisplayBBoxToPdf(display, geometry, { scale: 1 }); + for (const key of ['x0', 'y0', 'x1', 'y1']) assert.ok(Math.abs(roundtrip[key] - word.bbox[key]) < 0.01); + } + const pages = []; + const receipt = await extractPdfPages(bytes, { binaryPath, enableOcr: true, onPage: page => pages.push(page) }); + assert.equal(receipt.complete, true); + assertNative(pages[0], name); + assertLocated(pages[0], 'VECTOR EVIDENCE'); + } +}); diff --git a/src/adaptive.zig b/src/adaptive.zig index 6701c8d..3d7239d 100644 --- a/src/adaptive.zig +++ b/src/adaptive.zig @@ -15,6 +15,8 @@ const reading_order_regions = @import("reading_order_regions.zig"); const schema = @import("schema.zig"); const specialist_protocol = @import("specialist_protocol.zig"); const specialists = @import("specialists.zig"); +const page_geometry = @import("page_geometry.zig"); +const vector_routing = @import("vector_routing.zig"); pub const BBox = layout.BBox; pub const BlockKind = layout.BlockKind; @@ -88,6 +90,8 @@ pub const RouteReason = enum(u5) { layout_route_stub, table_route_stub, formula_route_stub, + unexplained_vector, + vector_evidence_limited, }; pub const RouteReasonMask = u32; @@ -122,6 +126,8 @@ pub const OcrAttempt = struct { crop_bbox: ?BBox = null, crop_image_sha256: ?[32]u8 = null, crop_image_byte_length: ?usize = null, + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, status: ocr.AttemptStatus, page_bbox: BBox = .{}, failure_stage: ?ocr.AttemptStage = null, @@ -150,6 +156,8 @@ pub const OcrAttempt = struct { pub const PageRoute = struct { page_index: u32, + page_geometry: ?page_geometry.PageGeometry = null, + vector_evidence: ?vector_routing.Summary = null, bbox: BBox, span_count: usize, image_count: usize, @@ -187,6 +195,7 @@ pub const RegionRoute = struct { page_index: u32, region_index: u32, layout_block_index: ?u32 = null, + vector_evidence: ?vector_routing.Candidate = null, bbox: BBox, block_kind: ?BlockKind = null, span_count: usize, @@ -290,6 +299,8 @@ pub const FormulaAttempt = struct { crop_dpi: ?u32 = null, crop_pixel_width: ?u32 = null, crop_pixel_height: ?u32 = null, + crop_pixel_to_pdf: ?page_geometry.Matrix = null, + crop_rasterizer_dpi: ?f64 = null, }; pub const FormulaArtifact = struct { @@ -595,6 +606,22 @@ pub fn extractDocument( const page_score = complexity.scorePage(page_input); var page_route = PageRoute.fromScore(page_score); + if (@hasDecl(@TypeOf(document.*), "getPageGeometry")) page_route.page_geometry = document.getPageGeometry(page_idx); + var vector_result: ?vector_routing.Result = null; + defer if (vector_result) |*value| value.deinit(); + if (@hasDecl(@TypeOf(document.*), "getPageVectorEvidenceFromContent")) { + var evidence = try document.getPageVectorEvidenceFromContent(page_idx, allocator, content_scratch_allocator, page_content); + defer evidence.deinit(); + vector_result = try vector_routing.classify(allocator, &evidence, spans, page_bbox, page_route.page_geometry); + page_route.vector_evidence = vector_result.?.summary; + if (vector_result.?.candidates.len > 0) { + page_route.route.native_fast_path = false; + page_route.reason_mask &= ~reasonBit(.native_fast_path); + page_route.reason_mask |= reasonBit(.unexplained_vector); + } + if (evidence.status != .complete or vector_result.?.summary.status != .complete) + page_route.reason_mask |= reasonBit(.vector_evidence_limited); + } if (!native_result.quality.quality_pass) suppressLayoutSpecialists(&page_route.route, &page_route.reason_mask); try page_routes.append(allocator, page_route); try trace_records.append(allocator, .{ @@ -618,7 +645,8 @@ pub fn extractDocument( // Image routes precede recognition so attempts retain the exact routed // region identity even when native text makes the page score look good. const image_route_start = region_routes.items.len; - const regional_ocr = usesRegionalOcr(page_route, native_result.quality.quality_pass); + const regional_ocr = usesRegionalOcr(page_route, native_result.quality.quality_pass) or + (vector_result != null and vector_result.?.candidates.len > 0 and !page_route.route.needs_ocr); if (regional_ocr) { for (image_boxes) |image| { const score = complexity.scoreRegion(page_input, image.bbox); @@ -627,12 +655,37 @@ pub fn extractDocument( try appendRouteTraces(allocator, &trace_records, page_index, region_index, route.route, route.reason_mask, route.span_count, 0); region_index += 1; } + if (vector_result) |value| { + for (value.candidates) |candidate| { + var covered = false; + for (region_routes.items[image_route_start..]) |*existing| { + if (!existing.route.needs_ocr or existing.bbox.x0 > candidate.bbox.x0 or existing.bbox.y0 > candidate.bbox.y0 or + existing.bbox.x1 < candidate.bbox.x1 or existing.bbox.y1 < candidate.bbox.y1) continue; + existing.vector_evidence = if (existing.vector_evidence) |old| + vector_routing.mergeCandidates(old, candidate, spans) orelse continue + else + candidate; + existing.reason_mask |= reasonBit(.unexplained_vector); + covered = true; + break; + } + if (covered) continue; + var route = RegionRoute.fromScore(.{ .region_index = region_index, .score = complexity.scoreRegion(page_input, candidate.bbox) }); + route.vector_evidence = candidate; + route.route = .{ .native_fast_path = false, .needs_ocr = true, .max_signal = 1 }; + route.reason_mask = reasonBit(.unexplained_vector) | reasonBit(.ocr_route_stub); + try region_routes.append(allocator, route); + try appendRouteTraces(allocator, &trace_records, page_index, region_index, route.route, route.reason_mask, route.span_count, 0); + region_index += 1; + } + } } const image_route_end = region_routes.items.len; var page_ocr_spans: ?[]layout.TextSpan = null; var page_attempt_count: usize = 0; if (!regional_ocr and page_route.route.needs_ocr) { - const selected = try recognizeOcrTarget(document, allocator, &ocr_attempts, options, page_idx, page_bbox, null, spans, image_boxes, &page_attempt_count); + const target_images = if (vector_result != null and vector_result.?.candidates.len > 0) &.{} else image_boxes; + const selected = try recognizeOcrTarget(document, allocator, &ocr_attempts, options, page_idx, page_bbox, null, spans, target_images, &page_attempt_count); if (selected) |owned| { errdefer ocr.freeSpans(allocator, owned); try ocr_pages.append(allocator, owned); @@ -658,7 +711,8 @@ pub fn extractDocument( continue; } requested_regions += 1; - if (try recognizeOcrTarget(document, allocator, &ocr_attempts, options, page_idx, route.bbox, route.region_index, spans, image_boxes, &page_attempt_count)) |owned| { + const target_images = if (route.vector_evidence != null) &.{} else image_boxes; + if (try recognizeOcrTarget(document, allocator, &ocr_attempts, options, page_idx, route.bbox, route.region_index, spans, target_images, &page_attempt_count)) |owned| { // Transfer strings only after capacity succeeds; free the // temporary array while combined owns its span strings. errdefer ocr.freeSpans(allocator, owned); @@ -836,13 +890,21 @@ pub fn extractDocument( region_summaries_transferred = true; } - for (page_layout.tables) |table| { - try tables.ensureUnusedCapacity(allocator, 1); - tables.appendAssumeCapacity(try copyTableGrid(allocator, table)); + // Axis-aligned table inference uses raw PDF coordinates. On a rotated + // OCR page those axes differ from the recognizer's rows and columns. + // Retain its authoritative word order until table inference can share + // the display transform; candidate routing may still describe evidence. + const promote_tables = page_ocr_spans == null or + (page_route.page_geometry != null and page_route.page_geometry.?.rotation == .none); + if (promote_tables) { + for (page_layout.tables) |table| { + try tables.ensureUnusedCapacity(allocator, 1); + tables.appendAssumeCapacity(try copyTableGrid(allocator, table)); + } } var filtered_ocr_layer_added = false; - if (page_layout.tables.len > 0 and page_view_quality_pass) { + if (promote_tables and page_layout.tables.len > 0 and page_view_quality_pass) { try layout_pages.ensureUnusedCapacity(allocator, 1); const layout_spans = try buildLayoutLayerSpans(allocator, &page_layout, applied_projected_order); if (layout_spans.len > 0) { @@ -1101,7 +1163,14 @@ fn executeFormulaSpecialists( defer crop_outcome.deinit(allocator); const crop_input = crop_outcome.input; var execution = if (crop_input) |crop| blk: { - const request_jsonl = try specialist_protocol.renderRegionRequestJsonl(allocator, result, context, route_index, crop.image_path); + var crop_context = context; + crop_context.crop_geometry = .{ + .pixel_to_pdf = crop.pixel_to_pdf, + .rasterizer_dpi = crop.rasterizer_dpi, + .pixel_width = crop.pixel_width, + .pixel_height = crop.pixel_height, + }; + const request_jsonl = try specialist_protocol.renderRegionRequestJsonl(allocator, result, crop_context, route_index, crop.image_path); defer allocator.free(request_jsonl); var completed = try specialists.runFormulaJsonl( allocator, @@ -1174,6 +1243,8 @@ fn executeFormulaSpecialists( .crop_dpi = if (crop_input) |crop| crop.dpi else null, .crop_pixel_width = if (crop_input) |crop| crop.pixel_width else null, .crop_pixel_height = if (crop_input) |crop| crop.pixel_height else null, + .crop_pixel_to_pdf = if (crop_input) |crop| crop.pixel_to_pdf else null, + .crop_rasterizer_dpi = if (crop_input) |crop| crop.rasterizer_dpi else null, }); } @@ -1619,6 +1690,8 @@ fn recognizeOcrTarget( .duration_ms = raster_outcome.duration_ms + recognition.duration_ms, .pixel_width = ocr_input.pixel_width, .pixel_height = ocr_input.pixel_height, + .pixel_to_pdf = ocr_input.pixel_to_pdf, + .rasterizer_dpi = ocr_input.rasterizer_dpi, .crop_bbox = if (region_id != null) ocr_input.pdf_bbox else null, .crop_image_sha256 = if (region_id != null) ocr_input.image_sha256 else null, .crop_image_byte_length = if (region_id != null) ocr_input.image_byte_length else null, @@ -1642,6 +1715,8 @@ fn recognizeOcrTarget( .duration_ms = raster_outcome.duration_ms + recognition.duration_ms, .pixel_width = ocr_input.pixel_width, .pixel_height = ocr_input.pixel_height, + .pixel_to_pdf = ocr_input.pixel_to_pdf, + .rasterizer_dpi = ocr_input.rasterizer_dpi, .crop_bbox = if (region_id != null) ocr_input.pdf_bbox else null, .crop_image_sha256 = if (region_id != null) ocr_input.image_sha256 else null, .crop_image_byte_length = if (region_id != null) ocr_input.image_byte_length else null, @@ -2518,6 +2593,8 @@ fn reasonName(reason: RouteReason) []const u8 { .layout_route_stub => "layout_route_stub", .table_route_stub => "table_route_stub", .formula_route_stub => "formula_route_stub", + .unexplained_vector => "unexplained_vector", + .vector_evidence_limited => "vector_evidence_limited", }; } @@ -2592,6 +2669,8 @@ const AppendOcrAttemptOptions = struct { duration_ms: u64 = 0, pixel_width: u32 = 0, pixel_height: u32 = 0, + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, spans: []const layout.TextSpan = &.{}, exit_code: ?u8 = null, diagnostic_code: ?ocr.DiagnosticCode = null, @@ -2644,6 +2723,8 @@ fn appendOcrAttempt( .duration_ms = options.duration_ms, .pixel_width = options.pixel_width, .pixel_height = options.pixel_height, + .pixel_to_pdf = options.pixel_to_pdf, + .rasterizer_dpi = options.rasterizer_dpi, .span_count = options.spans.len, .character_count = character_count, .mean_confidence = if (options.spans.len > 0) @floatCast(confidence_total / @as(f64, @floatFromInt(options.spans.len))) else 0, diff --git a/src/coverage.zig b/src/coverage.zig index 2fc7165..997d53d 100644 --- a/src/coverage.zig +++ b/src/coverage.zig @@ -28,7 +28,13 @@ pub fn assessRegion(result: anytype, route: anytype) Assessment { assessment.requested_recovery = true; const state = ocrState(result, route.page_index, route.region_index, route.bbox); assessment.recovery_state = combine(assessment.recovery_state, state); - unresolved = state != .completed or countRecognized(result, route.page_index, route.bbox, .fresh_ocr) == 0; + // A reused bitmap target can be larger than its unexplained vector + // candidate. Output elsewhere in that crop does not recover the candidate. + const recognized_bbox = if (@hasField(@TypeOf(route), "vector_evidence")) + if (route.vector_evidence) |candidate| candidate.bbox else route.bbox + else + route.bbox; + unresolved = state != .completed or countRecognized(result, route.page_index, recognized_bbox, .fresh_ocr) == 0; } if (route.route.needs_formula_model) { assessment.requested_recovery = true; @@ -190,6 +196,7 @@ const TestRoute = struct { region_index: u32 = 0, bbox: layout.BBox = .{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 100 }, route: struct { needs_ocr: bool = false, needs_formula_model: bool = false, needs_table_model: bool = false } = .{}, + vector_evidence: ?struct { bbox: layout.BBox } = null, }; const TestOcrStatus = enum { not_invoked, completed, empty, unavailable, failed, timeout, invalid_output }; const TestOcrAttempt = struct { page_index: u32 = 0, region_index: ?u32 = null, crop_bbox: ?layout.BBox = null, status: TestOcrStatus, selected: bool = false }; @@ -230,6 +237,27 @@ test "regional OCR recovery scopes include derived regions but exclude disjoint try std.testing.expectEqual(@as(usize, 1), assessPage(result, 0).unresolved_region_count); } +test "reused OCR crops require recognized text within the vector candidate" { + const candidate_box = layout.BBox{ .x0 = 20, .y0 = 20, .x1 = 40, .y1 = 40 }; + const route = TestRoute{ .route = .{ .needs_ocr = true }, .vector_evidence = .{ .bbox = candidate_box } }; + const disjoint = testSpan(.fresh_ocr, .{ .x0 = 50, .y0 = 50, .x1 = 60, .y1 = 60 }); + var result = TestResult{ + .region_routes = &.{route}, + .ocr_attempts = &.{.{ .region_index = 0, .status = .completed, .selected = true }}, + .reconciled = .{ .spans = &.{disjoint} }, + }; + const unresolved = assessRegion(result, route); + try std.testing.expectEqual(@as(usize, 1), unresolved.recognized_span_count); + try std.testing.expectEqual(@as(usize, 1), unresolved.unresolved_region_count); + try std.testing.expectEqual(Status.recovery_required, unresolved.status); + try std.testing.expectEqual(Status.recovery_required, assessPage(result, 0).status); + const overlapping = testSpan(.fresh_ocr, candidate_box); + result.reconciled.spans = &.{overlapping}; + try std.testing.expectEqual(Status.recovered, assessRegion(result, route).status); + try std.testing.expectEqual(Status.recovered, assessPage(result, 0).status); + try std.testing.expectEqual(Completeness.unverified, assessPage(result, 0).completeness); +} + test "coverage retains nonpainting clipped and uncertain evidence counts" { var hidden = testSpan(.native_pdf, .{ .x1 = 10, .y1 = 10 }); hidden.span.paint = .{ .rendering_mode = 3, .paint_state = .non_painting, .clip_state = .unclipped }; diff --git a/src/eval_runner.zig b/src/eval_runner.zig index 98b2270..614557e 100644 --- a/src/eval_runner.zig +++ b/src/eval_runner.zig @@ -2249,8 +2249,23 @@ test "eval runner adaptive mode emits OCR text metrics and counters" { const fake_rasterizer = \\#!/bin/sh \\last="" - \\for arg do last="$arg"; done - \\printf '\211PNG\r\n\032\n\000\000\000\rIHDR\000\000\003\350\000\000\007\320' > "$last.png" + \\previous="" + \\for arg do + \\ case "$previous" in -W) width="$arg" ;; -H) height="$arg" ;; esac + \\ previous="$arg" + \\ last="$arg" + \\done + \\emit_u32() { + \\ for bit_shift in 24 16 8 0; do + \\ octal=$(printf '%03o' "$((($1 >> bit_shift) & 255))") + \\ printf '%b' "\\$octal" + \\ done + \\} + \\{ + \\ printf '\211PNG\r\n\032\n\000\000\000\rIHDR' + \\ emit_u32 "$width" + \\ emit_u32 "$height" + \\} > "$last.png" \\ ; var raster_buf: [96]u8 = undefined; diff --git a/src/integration_test.zig b/src/integration_test.zig index f2d47a8..f823d76 100644 --- a/src/integration_test.zig +++ b/src/integration_test.zig @@ -353,8 +353,23 @@ test "adaptive OCR route invokes rasterizer and Tesseract adapter into fresh OCR const fake_rasterizer = \\#!/bin/sh \\last="" - \\for arg do last="$arg"; done - \\printf '\211PNG\r\n\032\n\000\000\000\rIHDR\000\000\003\350\000\000\007\320' > "$last.png" + \\previous="" + \\for arg do + \\ case "$previous" in -W) width="$arg" ;; -H) height="$arg" ;; esac + \\ previous="$arg" + \\ last="$arg" + \\done + \\emit_u32() { + \\ for bit_shift in 24 16 8 0; do + \\ octal=$(printf '%03o' "$((($1 >> bit_shift) & 255))") + \\ printf '%b' "\\$octal" + \\ done + \\} + \\{ + \\ printf '\211PNG\r\n\032\n\000\000\000\rIHDR' + \\ emit_u32 "$width" + \\ emit_u32 "$height" + \\} > "$last.png" \\ ; var raster_buf: [96]u8 = undefined; @@ -519,8 +534,23 @@ test "bounded OCR fallback records both attempts and selects higher quality resu const fake_rasterizer = \\#!/bin/sh \\last="" - \\for arg do last="$arg"; done - \\printf '\211PNG\r\n\032\n\000\000\000\rIHDR\000\000\003\350\000\000\007\320' > "$last.png" + \\previous="" + \\for arg do + \\ case "$previous" in -W) width="$arg" ;; -H) height="$arg" ;; esac + \\ previous="$arg" + \\ last="$arg" + \\done + \\emit_u32() { + \\ for bit_shift in 24 16 8 0; do + \\ octal=$(printf '%03o' "$((($1 >> bit_shift) & 255))") + \\ printf '%b' "\\$octal" + \\ done + \\} + \\{ + \\ printf '\211PNG\r\n\032\n\000\000\000\rIHDR' + \\ emit_u32 "$width" + \\ emit_u32 "$height" + \\} > "$last.png" \\ ; var raster_buf: [112]u8 = undefined; @@ -698,7 +728,7 @@ test "versioned schema renders native document manifest spans blocks chunks and const parsed = try std.json.parseFromSlice(std.json.Value, allocator, json, .{}); defer parsed.deinit(); try std.testing.expectEqualStrings("document_manifest", parsed.value.object.get("schema_name").?.string); - try std.testing.expectEqualStrings("0.17.0", parsed.value.object.get("schema_version").?.string); + try std.testing.expectEqualStrings("0.19.0", parsed.value.object.get("schema_version").?.string); try std.testing.expectEqualStrings("0.4.0-dev", parsed.value.object.get("parser_version").?.string); try std.testing.expectEqualStrings("document_manifest", parsed.value.object.get("record_type").?.string); try std.testing.expectEqualStrings("external-clean-native", parsed.value.object.get("source_id").?.string); @@ -744,7 +774,7 @@ test "versioned schema renders native document manifest spans blocks chunks and const debug_assets = parsed.value.object.get("debug_assets").?.array.items; try std.testing.expect(debug_assets.len > 0); try expectProvenanceObject(debug_assets[0]); - try std.testing.expectEqualStrings("0.17.0", debug_assets[0].object.get("schema_version").?.string); + try std.testing.expectEqualStrings("0.19.0", debug_assets[0].object.get("schema_version").?.string); try std.testing.expect(debug_assets[0].object.get("asset_kind") != null); try std.testing.expect(debug_assets[0].object.get("path") != null); try std.testing.expectEqual(.null, debug_assets[0].object.get("path").?); @@ -755,7 +785,7 @@ test "versioned schema renders native document manifest spans blocks chunks and try std.testing.expectEqualStrings("debug", debug_assets[0].object.get("provenance").?.object.get("source_kind").?.string); try std.testing.expect(std.mem.indexOf(u8, json, "\"schema_name\":\"document_manifest\"") != null); - try std.testing.expect(std.mem.indexOf(u8, json, "\"schema_version\":\"0.17.0\"") != null); + try std.testing.expect(std.mem.indexOf(u8, json, "\"schema_version\":\"0.19.0\"") != null); try std.testing.expect(std.mem.indexOf(u8, json, "\"record_type\":\"span\"") != null); try std.testing.expect(std.mem.indexOf(u8, json, "\"record_type\":\"block\"") != null); try std.testing.expect(std.mem.indexOf(u8, json, "\"record_type\":\"rag_chunk\"") != null); @@ -879,7 +909,7 @@ test "versioned artifact jsonl starts with manifest then typed records" { try std.testing.expect(manifest.value.object.get("output_artifacts") != null); try std.testing.expect(manifest.value.object.get("extraction_counts") != null); try std.testing.expect(manifest.value.object.get("capability_coverage") != null); - try std.testing.expect(std.mem.indexOf(u8, jsonl[first_newline + 1 ..], "\"schema_version\":\"0.17.0\"") != null); + try std.testing.expect(std.mem.indexOf(u8, jsonl[first_newline + 1 ..], "\"schema_version\":\"0.19.0\"") != null); try std.testing.expect(std.mem.indexOf(u8, jsonl[first_newline + 1 ..], "\"record_type\":\"span\"") != null); try std.testing.expect(std.mem.indexOf(u8, jsonl, "\"record_type\":\"route_trace\"") != null); try expectJsonlLinesParse(allocator, jsonl); @@ -911,7 +941,7 @@ test "specialist protocol emits batch request records for routed table formula r const requests = parsed.value.object.get("specialist_requests").?.array.items; try std.testing.expect(requests.len > 0); try expectProvenanceObject(requests[0]); - try std.testing.expectEqualStrings("0.17.0", requests[0].object.get("schema_version").?.string); + try std.testing.expectEqualStrings("0.19.0", requests[0].object.get("schema_version").?.string); try std.testing.expectEqualStrings("specialist_request", requests[0].object.get("record_type").?.string); try std.testing.expectEqualStrings("external-specialist", requests[0].object.get("source_id").?.string); try std.testing.expect(requests[0].object.get("requested_kind") != null); @@ -957,8 +987,23 @@ test "configured formula specialist produces owned linked lifecycle records" { const fake_rasterizer = \\#!/bin/sh \\last="" - \\for arg do last="$arg"; done - \\printf '\211PNG\r\n\032\n\000\000\000\rIHDR\000\000\000\310\000\000\000\144' > "$last.png" + \\previous="" + \\for arg do + \\ case "$previous" in -W) width="$arg" ;; -H) height="$arg" ;; esac + \\ previous="$arg" + \\ last="$arg" + \\done + \\emit_u32() { + \\ for bit_shift in 24 16 8 0; do + \\ octal=$(printf '%03o' "$((($1 >> bit_shift) & 255))") + \\ printf '%b' "\\$octal" + \\ done + \\} + \\{ + \\ printf '\211PNG\r\n\032\n\000\000\000\rIHDR' + \\ emit_u32 "$width" + \\ emit_u32 "$height" + \\} > "$last.png" \\ ; var raster_buf: [128]u8 = undefined; @@ -990,7 +1035,7 @@ test "configured formula specialist produces owned linked lifecycle records" { \\request_id=$(printf '%s' "$request" | sed -n 's/.*"request_id":"\([^"]*\)".*/\1/p') \\crop=$(printf '%s' "$request" | sed -n 's/.*"crop_image_path":"\([^"]*\)".*/\1/p') \\test -n "$crop" && test -f "$crop" || exit 7 - \\printf '{"schema_version":"0.17.0","record_type":"specialist_response","request_id":"%s","specialist_id":"fixture-formula","specialist_kind":"formula","status":"completed","formulas":[{"text":"x^2+y^2=z^2","format":"latex","confidence":0.95}]}\n' "$request_id" + \\printf '{"schema_version":"0.19.0","record_type":"specialist_response","request_id":"%s","specialist_id":"fixture-formula","specialist_kind":"formula","status":"completed","formulas":[{"text":"x^2+y^2=z^2","format":"latex","confidence":0.95}]}\n' "$request_id" \\ ); runtime.closeFile(script); @@ -1066,7 +1111,7 @@ test "configured formula specialist produces owned linked lifecycle records" { const formulas = response.object.get("formulas").?.array.items; try std.testing.expectEqual(@as(usize, 1), formulas.len); try std.testing.expectEqualStrings("formula", formulas[0].object.get("schema_name").?.string); - try std.testing.expectEqualStrings("0.17.0", formulas[0].object.get("schema_version").?.string); + try std.testing.expectEqualStrings("0.19.0", formulas[0].object.get("schema_version").?.string); try std.testing.expectEqualStrings("formula", formulas[0].object.get("record_type").?.string); try std.testing.expectEqualStrings("source-formula", formulas[0].object.get("source_id").?.string); try expectProvenanceObject(formulas[0]); @@ -1184,7 +1229,7 @@ test "batch and multi-page streaming specialist request ids are unique and equal } } -test "formula crop reports rotated pages as explicit unsupported outcomes" { +test "formula crop reports non-quarter rotations as explicit unsupported outcomes" { const allocator = std.testing.allocator; var threaded: std.Io.Threaded = .init(allocator, .{}); defer threaded.deinit(); @@ -1201,6 +1246,7 @@ test "formula crop reports rotated pages as explicit unsupported outcomes" { const doc = try zpdf.Document.openWithConfig(allocator, path, zpdf.ErrorConfig.permissive()); defer doc.close(); + doc.pages.items[0].rotation = 45; var outcome = try doc.rasterizeFormulaRegionDetailed( allocator, 0, @@ -1466,7 +1512,7 @@ test "versioned schema exposes financial table cell span metadata" { const tables = parsed.value.object.get("tables").?.array.items; try std.testing.expect(tables.len > 0); try expectProvenanceObject(tables[0]); - try std.testing.expectEqualStrings("0.17.0", tables[0].object.get("schema_version").?.string); + try std.testing.expectEqualStrings("0.19.0", tables[0].object.get("schema_version").?.string); try std.testing.expect(tables[0].object.get("logical_table_id") != null); try std.testing.expect(tables[0].object.get("table_part_index") != null); try std.testing.expect(tables[0].object.get("continued_from_table_id") != null); @@ -1476,7 +1522,7 @@ test "versioned schema exposes financial table cell span metadata" { const cells = tables[0].object.get("rows").?.array.items[0].object.get("cells").?.array.items; try std.testing.expect(cells.len > 0); try std.testing.expectEqualStrings("table_cell", cells[0].object.get("schema_name").?.string); - try std.testing.expectEqualStrings("0.17.0", cells[0].object.get("schema_version").?.string); + try std.testing.expectEqualStrings("0.19.0", cells[0].object.get("schema_version").?.string); try std.testing.expect(cells[0].object.get("cell_id") != null); try std.testing.expectEqualStrings("external-merged-cells", cells[0].object.get("source_id").?.string); try std.testing.expect(cells[0].object.get("raw_text") != null); diff --git a/src/main.zig b/src/main.zig index 2065c19..2c13e91 100644 --- a/src/main.zig +++ b/src/main.zig @@ -2039,8 +2039,23 @@ test "adaptive CLI emits specialist request JSONL sidecars" { try runtime.writeAllFile(raster_file, \\#!/bin/sh \\last="" - \\for arg do last="$arg"; done - \\printf '\211PNG\r\n\032\n\000\000\000\rIHDR\000\000\000\310\000\000\000\144' > "$last.png" + \\previous="" + \\for arg do + \\ case "$previous" in -W) width="$arg" ;; -H) height="$arg" ;; esac + \\ previous="$arg" + \\ last="$arg" + \\done + \\emit_u32() { + \\ for bit_shift in 24 16 8 0; do + \\ octal=$(printf '%03o' "$((($1 >> bit_shift) & 255))") + \\ printf '%b' "\\$octal" + \\ done + \\} + \\{ + \\ printf '\211PNG\r\n\032\n\000\000\000\rIHDR' + \\ emit_u32 "$width" + \\ emit_u32 "$height" + \\} > "$last.png" \\ ); runtime.closeFile(raster_file); @@ -2057,7 +2072,7 @@ test "adaptive CLI emits specialist request JSONL sidecars" { \\request_id=$(printf '%s' "$request" | sed -n 's/.*"request_id":"\([^"]*\)".*/\1/p') \\crop=$(printf '%s' "$request" | sed -n 's/.*"crop_image_path":"\([^"]*\)".*/\1/p') \\test -n "$crop" && test -f "$crop" || exit 7 - \\printf '{"schema_version":"0.17.0","record_type":"specialist_response","request_id":"%s","specialist_id":"cli-formula","specialist_kind":"formula","status":"completed","formulas":[{"text":"x^2+y^2=z^2","format":"latex","confidence":0.95}]}\n' "$request_id" + \\printf '{"schema_version":"0.19.0","record_type":"specialist_response","request_id":"%s","specialist_id":"cli-formula","specialist_kind":"formula","status":"completed","formulas":[{"text":"x^2+y^2=z^2","format":"latex","confidence":0.95}]}\n' "$request_id" \\ ); runtime.closeFile(formula_file); @@ -2131,7 +2146,7 @@ test "adaptive CLI emits specialist request JSONL sidecars" { const requests = try runtime.readFileAllocAlignedCwd(allocator, requests_path, .fromByteUnits(1)); defer allocator.free(requests); try std.testing.expect(std.mem.indexOf(u8, requests, "\"record_type\":\"specialist_request\"") != null); - try std.testing.expect(std.mem.indexOf(u8, requests, "\"schema_version\":\"0.17.0\"") != null); + try std.testing.expect(std.mem.indexOf(u8, requests, "\"schema_version\":\"0.19.0\"") != null); try std.testing.expect(std.mem.indexOf(u8, requests, "\"source_id\":\"external-specialist-cli\"") != null); try std.testing.expect(std.mem.indexOf(u8, requests, "\"requested_kind\":\"") != null); try std.testing.expect(std.mem.indexOf(u8, requests, "\"requested_outputs\"") != null); diff --git a/src/ocr.zig b/src/ocr.zig index ab396de..c21ce1d 100644 --- a/src/ocr.zig +++ b/src/ocr.zig @@ -64,6 +64,24 @@ pub fn recognizeRegionDetailed( .duration_ms = elapsedMs(started_ns), .diagnostic_code = .tesseract_c_backend_disabled, }; + switch (@as(anyerror, err)) { + error.EmptyImage, + error.InvalidBox, + error.InvalidMatrix, + error.SingularMatrix, + error.InvalidPoint, + error.InvalidDpi, + error.InvalidRasterDimensions, + error.InvalidWordBounds, + error.InvalidConfidence, + error.ImageReadFailed, + => break :blk .{ + .status = .invalid_output, + .duration_ms = elapsedMs(started_ns), + .diagnostic_code = .invalid_tesseract_output, + }, + else => {}, + } return err; }; break :blk .{ diff --git a/src/ocr/crop.zig b/src/ocr/crop.zig index 124375c..64e7f1f 100644 --- a/src/ocr/crop.zig +++ b/src/ocr/crop.zig @@ -1,6 +1,7 @@ //! Pixel-aligned PDF crop geometry and bounded raster identity. const std = @import("std"); const layout = @import("../layout.zig"); +const page_geometry = @import("../page_geometry.zig"); const runtime = @import("../runtime.zig"); pub const Geometry = struct { @@ -12,32 +13,14 @@ pub const Geometry = struct { }; pub fn geometry(page: layout.BBox, region: layout.BBox, padding: f64, dpi: u32) ?Geometry { - for ([_]f64{ page.x0, page.y0, page.x1, page.y1, region.x0, region.y0, region.x1, region.y1, padding }) |value| { - if (!std.math.isFinite(value)) return null; - } - if (dpi == 0 or padding < 0 or page.x1 <= page.x0 or page.y1 <= page.y0 or region.x1 <= region.x0 or region.y1 <= region.y0) return null; - const bbox = layout.BBox{ - .x0 = @max(page.x0, region.x0 - padding), - .y0 = @max(page.y0, region.y0 - padding), - .x1 = @min(page.x1, region.x1 + padding), - .y1 = @min(page.y1, region.y1 + padding), - }; - if (bbox.x1 <= bbox.x0 or bbox.y1 <= bbox.y0) return null; - const scale = @as(f64, @floatFromInt(dpi)) / 72.0; - const left = @floor((bbox.x0 - page.x0) * scale); - const top = @floor((page.y1 - bbox.y1) * scale); - const right = @ceil((bbox.x1 - page.x0) * scale); - const bottom = @ceil((page.y1 - bbox.y0) * scale); - const limit: f64 = @floatFromInt(std.math.maxInt(i32)); - if (right > limit or bottom > limit or right <= left or bottom <= top) return null; + const page_transform = page_geometry.PageGeometry.init(page, null, 0, 1) catch return null; + const plan = page_transform.planCrop(region, .{ .padding_points = padding, .dpi = @floatFromInt(dpi) }) catch return null; return .{ - // The raster is rounded outward; use its actual pixel boundaries for - // OCR coordinates rather than stretching words into the requested box. - .pdf_bbox = .{ .x0 = page.x0 + left / scale, .y0 = page.y1 - bottom / scale, .x1 = page.x0 + right / scale, .y1 = page.y1 - top / scale }, - .x = @intFromFloat(left), - .y = @intFromFloat(top), - .width = @intFromFloat(right - left), - .height = @intFromFloat(bottom - top), + .pdf_bbox = plan.pdf_bbox, + .x = plan.x, + .y = plan.y, + .width = plan.width, + .height = plan.height, }; } diff --git a/src/ocr/tesseract_c.zig b/src/ocr/tesseract_c.zig index 738dc61..c7e3cea 100644 --- a/src/ocr/tesseract_c.zig +++ b/src/ocr/tesseract_c.zig @@ -79,7 +79,8 @@ fn recognizeRegionEnabled( if (!options.enable_tesseract_c) unreachable; const c = enabled.c; - if (input.pixel_width == 0 or input.pixel_height == 0) return error.EmptyImage; + try input.validateGeometry(); + if (input.pixel_width > std.math.maxInt(c_int) or input.pixel_height > std.math.maxInt(c_int)) return error.InvalidRasterDimensions; const image_path_z = try allocator.dupeSentinel(u8, input.image_path, 0); defer allocator.free(image_path_z); @@ -98,6 +99,7 @@ fn recognizeRegionEnabled( var pix = c.pixRead(image_path_z.ptr); if (pix == null) return error.ImageReadFailed; defer c.pixDestroy(&pix); + if (c.pixGetWidth(pix.?) != input.pixel_width or c.pixGetHeight(pix.?) != input.pixel_height) return error.InvalidRasterDimensions; c.TessBaseAPISetImage2(api, pix.?); c.TessBaseAPISetSourceResolution(api, @intCast(config.dpi)); @@ -167,7 +169,8 @@ fn appendWordSpan( var right: c_int = 0; var bottom: c_int = 0; if (c.TessPageIteratorBoundingBox(page_iterator, c.RIL_WORD, &left, &top, &right, &bottom) == 0) return; - if (right <= left or bottom <= top or left < 0 or top < 0) return; + if (right <= left or bottom <= top or left < 0 or top < 0) return error.InvalidWordBounds; + const bbox = try input.mapWordBBox(@intCast(left), @intCast(top), @intCast(right - left), @intCast(bottom - top)); var is_bold: c_int = 0; var is_italic: c_int = 0; @@ -193,15 +196,10 @@ fn appendWordSpan( errdefer allocator.free(text); const confidence = c.TessResultIteratorConfidence(iterator, c.RIL_WORD); + if (!std.math.isFinite(confidence)) return error.InvalidConfidence; try spans.append(allocator, layout.TextSpan.init(.{ .page_index = input.page_index, - .bbox = mapPixelBBox( - input, - @intCast(left), - @intCast(top), - @intCast(right - left), - @intCast(bottom - top), - ), + .bbox = bbox, .text = text, .source = .fresh_ocr, .confidence = @max(0.0, @min(1.0, confidence / 100.0)), @@ -212,24 +210,39 @@ fn appendWordSpan( }, .block_id = block_id, .line_id = line_id, + .leading_boundary = wordBoundary(spans.items, block_id, line_id), })); } -fn mapPixelBBox(input: types.OcrInput, left: u32, top: u32, width: u32, height: u32) layout.BBox { - const pdf_width = input.pdf_bbox.x1 - input.pdf_bbox.x0; - const pdf_height = input.pdf_bbox.y1 - input.pdf_bbox.y0; - const x_scale = pdf_width / @as(f64, @floatFromInt(input.pixel_width)); - const y_scale = pdf_height / @as(f64, @floatFromInt(input.pixel_height)); - - const x0 = input.pdf_bbox.x0 + @as(f64, @floatFromInt(left)) * x_scale; - const x1 = x0 + @as(f64, @floatFromInt(width)) * x_scale; - const y1 = input.pdf_bbox.y1 - @as(f64, @floatFromInt(top)) * y_scale; - const y0 = y1 - @as(f64, @floatFromInt(height)) * y_scale; - - return .{ - .x0 = x0, - .y0 = y0, - .x1 = x1, - .y1 = y1, - }; +fn wordBoundary(spans: []const layout.TextSpan, block_id: u32, line_id: u32) layout.TextBoundary { + if (spans.len == 0) return .region_break; + const previous = spans[spans.len - 1]; + if (previous.block_id != block_id) return .block_break; + if (previous.line_id != line_id) return .line_break; + return .space; +} + +test "C word boundaries begin each accepted region and follow selected positions" { + try std.testing.expectEqual(layout.TextBoundary.region_break, wordBoundary(&.{}, 8, 3)); + const spans = [_]layout.TextSpan{layout.TextSpan.init(.{ + .bbox = .{ .x0 = 0, .y0 = 0, .x1 = 10, .y1 = 10 }, + .text = "previous accepted", + .block_id = 8, + .line_id = 3, + })}; + try std.testing.expectEqual(layout.TextBoundary.space, wordBoundary(&spans, 8, 3)); + try std.testing.expectEqual(layout.TextBoundary.line_break, wordBoundary(&spans, 8, 4)); + try std.testing.expectEqual(layout.TextBoundary.block_break, wordBoundary(&spans, 9, 4)); +} + +test "enabled C backend validates geometry before loading an image" { + if (!options.enable_tesseract_c) return error.SkipZigTest; + try std.testing.expectError(error.InvalidMatrix, recognizeRegion(std.testing.allocator, .{ + .page_index = 0, + .pdf_bbox = .{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 100 }, + .image_path = "unused.png", + .pixel_width = 100, + .pixel_height = 100, + .pixel_to_pdf = .{ 1, 0, 0, 1, std.math.nan(f64), 0 }, + }, .{ .backend = .tesseract_c })); } diff --git a/src/ocr/tesseract_cli.zig b/src/ocr/tesseract_cli.zig index a10dd4b..e1e8dc2 100644 --- a/src/ocr/tesseract_cli.zig +++ b/src/ocr/tesseract_cli.zig @@ -33,12 +33,12 @@ pub fn recognizeRegionDetailed( input: types.OcrInput, config: types.OcrConfig, ) !types.RecognitionOutcome { - if (input.pixel_width == 0 or input.pixel_height == 0) { + input.validateGeometry() catch { return .{ .status = .invalid_output, .diagnostic_code = .invalid_tesseract_output, }; - } + }; const psm_arg = try std.fmt.allocPrint(allocator, "{d}", .{config.psm.tesseractNumber()}); defer allocator.free(psm_arg); @@ -109,6 +109,8 @@ pub fn recognizeRegionDetailed( .pdf_bbox = input.pdf_bbox, .pixel_width = input.pixel_width, .pixel_height = input.pixel_height, + .pixel_to_pdf = input.pixel_to_pdf, + .rasterizer_dpi = input.rasterizer_dpi, }) catch |err| switch (err) { error.OutOfMemory => return err, else => return .{ @@ -243,3 +245,42 @@ fn writeExecutableScript(path: []const u8, contents: []const u8) !void { runtime.closeFile(file); try std.testing.expectEqual(@as(u8, 0), try runtime.runIgnored(&.{ "chmod", "+x", path })); } + +test "CLI forwards raster transform and refuses out of image OCR words" { + if (@import("builtin").os.tag == .windows) return error.SkipZigTest; + const allocator = std.testing.allocator; + var threaded: std.Io.Threaded = .init(allocator, .{}); + defer threaded.deinit(); + runtime.setIo(threaded.io()); + var path_buf: [128]u8 = undefined; + const path = try std.fmt.bufPrint(&path_buf, "pdf-parser-tesseract-matrix-{x}.sh", .{runtime.nanoTimestamp()}); + defer runtime.deleteFileCwd(path); + var exec_buf: [144]u8 = undefined; + const executable = try std.fmt.bufPrint(&exec_buf, "./{s}", .{path}); + const header = "#!/bin/sh\nprintf 'level\\tpage_num\\tblock_num\\tpar_num\\tline_num\\tword_num\\tleft\\ttop\\twidth\\theight\\tconf\\ttext\\n'\n"; + try writeExecutableScript(path, header ++ "printf '5\\t1\\t1\\t1\\t1\\t1\\t10\\t20\\t30\\t5\\t90\\tMapped\\n'\n"); + var input = testInput(); + input.pixel_to_pdf = .{ 0, -0.5, -0.5, 0, 200, 300 }; + input.rasterizer_dpi = 144; + var valid = try recognizeRegionDetailed(allocator, input, .{ .executable = executable }); + defer valid.deinit(allocator); + try std.testing.expectEqual(types.AttemptStatus.completed, valid.status); + try std.testing.expectEqual(@as(usize, 1), valid.spans.len); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 187.5, .y0 = 280, .x1 = 190, .y1 = 295 }, valid.spans[0].bbox); + try std.testing.expectEqual(layout.TextBoundary.region_break, valid.spans[0].leading_boundary.?); + try writeExecutableScript(path, header ++ "printf '5\\t1\\t1\\t1\\t1\\t1\\t90\\t20\\t30\\t5\\t90\\tOutside\\n'\n"); + var refused = try recognizeRegionDetailed(allocator, input, .{ .executable = executable }); + defer refused.deinit(allocator); + try std.testing.expectEqual(types.AttemptStatus.invalid_output, refused.status); + try std.testing.expectEqual(@as(usize, 0), refused.spans.len); + try std.testing.expectEqual(types.DiagnosticCode.invalid_tesseract_output, refused.diagnostic_code.?); +} + +test "CLI refuses invalid geometry before invoking Tesseract" { + var input = testInput(); + input.pixel_to_pdf = .{ 1, 0, 0, 1, std.math.nan(f64), 0 }; + var outcome = try recognizeRegionDetailed(std.testing.allocator, input, .{ .executable = "missing-tesseract-geometry-refusal" }); + defer outcome.deinit(std.testing.allocator); + try std.testing.expectEqual(types.AttemptStatus.invalid_output, outcome.status); + try std.testing.expectEqual(types.DiagnosticCode.invalid_tesseract_output, outcome.diagnostic_code.?); +} diff --git a/src/ocr/tesseract_tsv.zig b/src/ocr/tesseract_tsv.zig index eda01f4..28a6ad7 100644 --- a/src/ocr/tesseract_tsv.zig +++ b/src/ocr/tesseract_tsv.zig @@ -2,12 +2,16 @@ const std = @import("std"); const layout = @import("../layout.zig"); +const types = @import("types.zig"); +const page_geometry = @import("../page_geometry.zig"); pub const ParseInput = struct { page_index: u32, pdf_bbox: layout.BBox, pixel_width: u32, pixel_height: u32, + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, }; const expected_header = "level\tpage_num\tblock_num\tpar_num\tline_num\tword_num\tleft\ttop\twidth\theight\tconf\ttext"; @@ -17,7 +21,16 @@ pub fn parse( tsv: []const u8, input: ParseInput, ) ![]layout.TextSpan { - if (input.pixel_width == 0 or input.pixel_height == 0) return error.EmptyImage; + const geometry = types.OcrInput{ + .page_index = input.page_index, + .pdf_bbox = input.pdf_bbox, + .image_path = "", + .pixel_width = input.pixel_width, + .pixel_height = input.pixel_height, + .pixel_to_pdf = input.pixel_to_pdf, + .rasterizer_dpi = input.rasterizer_dpi, + }; + try geometry.validateGeometry(); var spans: std.ArrayList(layout.TextSpan) = .empty; errdefer { @@ -26,6 +39,7 @@ pub fn parse( } var seen_header = false; + var previous_word: ?Row = null; var lines = std.mem.splitScalar(u8, tsv, '\n'); while (lines.next()) |raw_line| { const line = std.mem.trimEnd(u8, raw_line, "\r"); @@ -39,13 +53,14 @@ pub fn parse( const row = try parseRow(line); if (row.level != 5 or row.confidence < 0 or row.text.len == 0) continue; + const bbox = try geometry.mapWordBBox(row.left, row.top, row.width, row.height); const text = try allocator.dupe(u8, row.text); errdefer allocator.free(text); try spans.append(allocator, layout.TextSpan.init(.{ .page_index = input.page_index, - .bbox = mapPixelBBox(input, row.left, row.top, row.width, row.height), + .bbox = bbox, .text = text, .source = .fresh_ocr, .confidence = @max(0.0, @min(1.0, row.confidence / 100.0)), @@ -56,7 +71,17 @@ pub fn parse( }, .block_id = row.block_num, .line_id = row.line_num, + .leading_boundary = if (previous_word) |previous| + if (row.page_num != previous.page_num or row.block_num != previous.block_num) + .block_break + else if (row.par_num != previous.par_num or row.line_num != previous.line_num) + .line_break + else + .space + else + .region_break, })); + previous_word = row; } if (!seen_header) return error.InvalidTsvHeader; @@ -65,8 +90,11 @@ pub fn parse( const Row = struct { level: u8, + page_num: u32, block_num: u32, + par_num: u32, line_num: u32, + word_num: u32, left: u32, top: u32, width: u32, @@ -85,15 +113,20 @@ fn parseRow(line: []const u8) !Row { } fields[11] = rest; + const confidence = std.fmt.parseFloat(f32, fields[10]) catch return error.InvalidConfidence; + if (!std.math.isFinite(confidence)) return error.InvalidConfidence; return .{ .level = try parseUnsigned(u8, fields[0]), + .page_num = try parseUnsigned(u32, fields[1]), .block_num = try parseUnsigned(u32, fields[2]), + .par_num = try parseUnsigned(u32, fields[3]), .line_num = try parseUnsigned(u32, fields[4]), + .word_num = try parseUnsigned(u32, fields[5]), .left = try parseUnsigned(u32, fields[6]), .top = try parseUnsigned(u32, fields[7]), .width = try parseUnsigned(u32, fields[8]), .height = try parseUnsigned(u32, fields[9]), - .confidence = std.fmt.parseFloat(f32, fields[10]) catch return error.InvalidConfidence, + .confidence = confidence, .text = fields[11], }; } @@ -102,25 +135,6 @@ fn parseUnsigned(comptime T: type, value: []const u8) !T { return std.fmt.parseInt(T, value, 10) catch return error.InvalidCoordinate; } -fn mapPixelBBox(input: ParseInput, left: u32, top: u32, width: u32, height: u32) layout.BBox { - const pdf_width = input.pdf_bbox.x1 - input.pdf_bbox.x0; - const pdf_height = input.pdf_bbox.y1 - input.pdf_bbox.y0; - const x_scale = pdf_width / @as(f64, @floatFromInt(input.pixel_width)); - const y_scale = pdf_height / @as(f64, @floatFromInt(input.pixel_height)); - - const x0 = input.pdf_bbox.x0 + @as(f64, @floatFromInt(left)) * x_scale; - const x1 = x0 + @as(f64, @floatFromInt(width)) * x_scale; - const y1 = input.pdf_bbox.y1 - @as(f64, @floatFromInt(top)) * y_scale; - const y0 = y1 - @as(f64, @floatFromInt(height)) * y_scale; - - return .{ - .x0 = x0, - .y0 = y0, - .x1 = x1, - .y1 = y1, - }; -} - fn freeTestSpans(spans: []layout.TextSpan) void { for (spans) |span| std.testing.allocator.free(@constCast(span.text)); std.testing.allocator.free(spans); @@ -177,3 +191,112 @@ test "parse skips empty and rejected word rows" { try std.testing.expectEqual(@as(usize, 0), spans.len); } + +fn mappingTestInput() ParseInput { + return .{ + .page_index = 3, + .pdf_bbox = .{ .x0 = 10, .y0 = 20, .x1 = 200, .y1 = 300 }, + .pixel_width = 100, + .pixel_height = 100, + }; +} + +test "TSV uses literal quarter turn and affine four corner pixel mapping" { + const row = expected_header ++ "\n5\t1\t1\t1\t1\t1\t10\t20\t30\t5\t90\tword\n"; + const cases = [_]struct { matrix: page_geometry.Matrix, expected: layout.BBox }{ + .{ .matrix = .{ 0, -0.5, -0.5, 0, 200, 300 }, .expected = .{ .x0 = 187.5, .y0 = 280, .x1 = 190, .y1 = 295 } }, + .{ .matrix = .{ -0.5, 0, 0, 0.5, 200, 20 }, .expected = .{ .x0 = 180, .y0 = 30, .x1 = 195, .y1 = 32.5 } }, + .{ .matrix = .{ 0, 0.5, 0.5, 0, 10, 20 }, .expected = .{ .x0 = 20, .y0 = 25, .x1 = 22.5, .y1 = 40 } }, + // A shear with opposite edge extrema requires all four corners. + .{ .matrix = .{ 1, 0.5, -1, 1, 10, 20 }, .expected = .{ .x0 = -5, .y0 = 45, .x1 = 30, .y1 = 65 } }, + }; + for (cases) |case| { + var input = mappingTestInput(); + input.pixel_to_pdf = case.matrix; + input.rasterizer_dpi = 144; + const spans = try parse(std.testing.allocator, row, input); + defer freeTestSpans(spans); + try std.testing.expectEqual(@as(usize, 1), spans.len); + try std.testing.expectEqualDeep(case.expected, spans[0].bbox); + try std.testing.expectEqual(layout.TextBoundary.region_break, spans[0].leading_boundary.?); + } +} + +test "TSV boundaries follow selected word identities and isolate independent regions" { + const tsv = expected_header ++ "\n" ++ + "5\t1\t8\t1\t1\t1\t0\t0\t10\t10\t-1\trejected\n" ++ + "5\t1\t8\t1\t1\t2\t0\t0\t10\t10\t90\t\n" ++ + "5\t1\t1\t1\t1\t1\t10\t20\t10\t5\t90\tA\n" ++ + "5\t1\t99\t1\t9\t1\t0\t0\t10\t10\t-1\tignored\n" ++ + "5\t1\t1\t1\t1\t2\t30\t20\t10\t5\t90\tB\n" ++ + "5\t1\t1\t1\t2\t1\t10\t40\t10\t5\t90\tC\n" ++ + "5\t1\t1\t2\t1\t1\t10\t60\t10\t5\t90\tD\n" ++ + "5\t1\t2\t1\t1\t1\t10\t80\t10\t5\t90\tE\n"; + var input = mappingTestInput(); + input.pixel_to_pdf = .{ 0, -0.5, -0.5, 0, 200, 300 }; + // Each OCR call begins a region even when its first selected IDs match + // those of another crop. Skipped words never determine a boundary. + for (0..2) |_| { + const spans = try parse(std.testing.allocator, tsv, input); + defer freeTestSpans(spans); + try std.testing.expectEqual(@as(usize, 5), spans.len); + for (spans, [_]layout.TextBoundary{ .region_break, .space, .line_break, .line_break, .block_break }) |span, expected| { + try std.testing.expectEqual(expected, span.leading_boundary.?); + } + } +} + +test "TSV refuses invalid image transform dpi and accepted word bounds" { + const row = expected_header ++ "\n5\t1\t1\t1\t1\t1\t10\t20\t30\t5\t90\tword\n"; + var input = mappingTestInput(); + input.pixel_width = 0; + try std.testing.expectError(error.EmptyImage, parse(std.testing.allocator, row, input)); + input = mappingTestInput(); + input.pixel_height = 0; + try std.testing.expectError(error.EmptyImage, parse(std.testing.allocator, row, input)); + input = mappingTestInput(); + input.pdf_bbox.x0 = std.math.nan(f64); + try std.testing.expectError(error.InvalidBox, parse(std.testing.allocator, row, input)); + input = mappingTestInput(); + input.pdf_bbox.x1 = input.pdf_bbox.x0; + try std.testing.expectError(error.InvalidBox, parse(std.testing.allocator, row, input)); + input = mappingTestInput(); + for ([_]page_geometry.Matrix{ .{ std.math.nan(f64), 0, 0, 1, 0, 0 }, .{ 1, 0, 0, 1, std.math.inf(f64), 0 } }) |matrix| { + input.pixel_to_pdf = matrix; + try std.testing.expectError(error.InvalidMatrix, parse(std.testing.allocator, row, input)); + } + input.pixel_to_pdf = .{ 0, 0, 0, 1, 0, 0 }; + try std.testing.expectError(error.SingularMatrix, parse(std.testing.allocator, row, input)); + input = mappingTestInput(); + for ([_]f64{ 0, -1, std.math.nan(f64), std.math.inf(f64) }) |dpi| { + input.rasterizer_dpi = dpi; + try std.testing.expectError(error.InvalidDpi, parse(std.testing.allocator, row, input)); + } + input = mappingTestInput(); + for ([_][]const u8{ + "5\t1\t1\t1\t1\t1\t90\t20\t30\t5\t90\tword\n", + "5\t1\t1\t1\t1\t1\t10\t99\t30\t5\t90\tword\n", + "5\t1\t1\t1\t1\t1\t10\t20\t0\t5\t90\tword\n", + "5\t1\t1\t1\t1\t1\t4294967295\t0\t4294967295\t5\t90\tword\n", + }) |bad_row| { + const data = try std.mem.concat(std.testing.allocator, u8, &.{ row, bad_row }); + defer std.testing.allocator.free(data); + // A later malformed word must release the earlier accepted span. + try std.testing.expectError(error.InvalidWordBounds, parse(std.testing.allocator, data, input)); + } +} + +test "TSV rejects malformed selected identities and nonfinite confidence" { + for ([_]struct { row: []const u8, failure: anyerror }{ + .{ .row = "5\t1\t1\tbad\t1\t1\t0\t0\t5\t5\t90\tword\n", .failure = error.InvalidCoordinate }, + .{ .row = "5\t1\t1\t1\t1\tbad\t0\t0\t5\t5\t90\tword\n", .failure = error.InvalidCoordinate }, + .{ .row = "5\t1\t1\t1\t1\t1\t-1\t0\t5\t5\t90\tword\n", .failure = error.InvalidCoordinate }, + .{ .row = "5\t1\t1\t1\t1\t1\t0\t0\t5\t5\tnan\tword\n", .failure = error.InvalidConfidence }, + .{ .row = "5\t1\t1\t1\t1\t1\t0\t0\t5\t5\tinf\tword\n", .failure = error.InvalidConfidence }, + .{ .row = "5\t1\t1\n", .failure = error.InvalidTsvRow }, + }) |case| { + const data = try std.mem.concat(std.testing.allocator, u8, &.{ expected_header, "\n", case.row }); + defer std.testing.allocator.free(data); + try std.testing.expectError(case.failure, parse(std.testing.allocator, data, mappingTestInput())); + } +} diff --git a/src/ocr/types.zig b/src/ocr/types.zig index 8af2ee2..316cc4a 100644 --- a/src/ocr/types.zig +++ b/src/ocr/types.zig @@ -1,4 +1,6 @@ +const std = @import("std"); const layout = @import("../layout.zig"); +const page_geometry = @import("../page_geometry.zig"); pub const Backend = enum { tesseract_cli, @@ -99,8 +101,60 @@ pub const OcrInput = struct { image_path: []const u8, pixel_width: u32, pixel_height: u32, + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, image_sha256: ?[32]u8 = null, image_byte_length: ?usize = null, + + pub fn validateGeometry(self: OcrInput) !void { + if (self.pixel_width == 0 or self.pixel_height == 0) return error.EmptyImage; + _ = try page_geometry.mapBBox(page_geometry.identity, self.pdf_bbox); + if (self.pdf_bbox.x1 <= self.pdf_bbox.x0 or self.pdf_bbox.y1 <= self.pdf_bbox.y0) return error.InvalidBox; + if (self.rasterizer_dpi) |dpi| { + if (!std.math.isFinite(dpi) or dpi <= 0) return error.InvalidDpi; + } + if (self.pixel_to_pdf) |matrix| { + _ = try page_geometry.inverse(matrix); + _ = try page_geometry.mapBBox(matrix, .{ + .x0 = 0, + .y0 = 0, + .x1 = @floatFromInt(self.pixel_width), + .y1 = @floatFromInt(self.pixel_height), + }); + } + } + + pub fn mapWordBBox(self: OcrInput, left: u32, top: u32, width: u32, height: u32) !layout.BBox { + try self.validateGeometry(); + const right = @as(u64, left) + width; + const bottom = @as(u64, top) + height; + if (width == 0 or height == 0 or right > self.pixel_width or bottom > self.pixel_height) return error.InvalidWordBounds; + if (self.pixel_to_pdf) |matrix| { + const bbox = try page_geometry.mapBBox(matrix, .{ + .x0 = @floatFromInt(left), + .y0 = @floatFromInt(top), + .x1 = @floatFromInt(right), + .y1 = @floatFromInt(bottom), + }); + if (bbox.x1 <= bbox.x0 or bbox.y1 <= bbox.y0) return error.InvalidWordBounds; + return bbox; + } + // Keep the original proportional calculation for callers without a + // raster transform, including its exact outward-crop bbox semantics. + const x_scale = (self.pdf_bbox.x1 - self.pdf_bbox.x0) / @as(f64, @floatFromInt(self.pixel_width)); + const y_scale = (self.pdf_bbox.y1 - self.pdf_bbox.y0) / @as(f64, @floatFromInt(self.pixel_height)); + const x0 = self.pdf_bbox.x0 + @as(f64, @floatFromInt(left)) * x_scale; + const y1 = self.pdf_bbox.y1 - @as(f64, @floatFromInt(top)) * y_scale; + const bbox = layout.BBox{ + .x0 = x0, + .y0 = y1 - @as(f64, @floatFromInt(height)) * y_scale, + .x1 = x0 + @as(f64, @floatFromInt(width)) * x_scale, + .y1 = y1, + }; + _ = try page_geometry.mapBBox(page_geometry.identity, bbox); + if (bbox.x1 <= bbox.x0 or bbox.y1 <= bbox.y0) return error.InvalidWordBounds; + return bbox; + } }; pub const RasterizeOutcome = struct { diff --git a/src/page_geometry.zig b/src/page_geometry.zig new file mode 100644 index 0000000..395b810 --- /dev/null +++ b/src/page_geometry.zig @@ -0,0 +1,505 @@ +const std = @import("std"); +const layout = @import("layout.zig"); + +pub const BBox = layout.BBox; +/// PDF affine convention: x' = a*x + c*y + e; y' = b*x + d*y + f. +pub const Matrix = [6]f64; +pub const Point = struct { x: f64, y: f64 }; +pub const identity: Matrix = .{ 1, 0, 0, 1, 0, 0 }; +pub const GeometryError = error{ + InvalidBox, + EmptyCropBox, + InvalidUserUnit, + UnsupportedRotation, + InvalidMatrix, + SingularMatrix, + InvalidPoint, + InvalidDpi, + InvalidPadding, + EmptyRegion, + RasterExtentOverflow, + InvalidRasterDimensions, +}; + +/// Apply inner first, then outer. Composition itself does not validate inputs. +pub fn concat(outer: Matrix, inner: Matrix) Matrix { + return .{ + outer[0] * inner[0] + outer[2] * inner[1], + outer[1] * inner[0] + outer[3] * inner[1], + outer[0] * inner[2] + outer[2] * inner[3], + outer[1] * inner[2] + outer[3] * inner[3], + outer[0] * inner[4] + outer[2] * inner[5] + outer[4], + outer[1] * inner[4] + outer[3] * inner[5] + outer[5], + }; +} + +pub fn inverse(matrix: Matrix) GeometryError!Matrix { + try validateMatrix(matrix); + const determinant = matrix[0] * matrix[3] - matrix[1] * matrix[2]; + if (!std.math.isFinite(determinant)) return error.InvalidMatrix; + if (determinant == 0) return error.SingularMatrix; + const result: Matrix = .{ + matrix[3] / determinant, + -matrix[1] / determinant, + -matrix[2] / determinant, + matrix[0] / determinant, + (matrix[2] * matrix[5] - matrix[3] * matrix[4]) / determinant, + (matrix[1] * matrix[4] - matrix[0] * matrix[5]) / determinant, + }; + try validateMatrix(result); + return result; +} + +pub fn mapPoint(matrix: Matrix, point: Point) GeometryError!Point { + try validateMatrix(matrix); + if (!std.math.isFinite(point.x) or !std.math.isFinite(point.y)) return error.InvalidPoint; + const result: Point = .{ + .x = matrix[0] * point.x + matrix[2] * point.y + matrix[4], + .y = matrix[1] * point.x + matrix[3] * point.y + matrix[5], + }; + if (!std.math.isFinite(result.x) or !std.math.isFinite(result.y)) return error.InvalidPoint; + return result; +} + +/// Map all four corners. Non-quarter Form matrices therefore produce an AABB. +pub fn mapBBox(matrix: Matrix, box: BBox) GeometryError!BBox { + try validateBox(box, false); + const corners = [_]Point{ + .{ .x = box.x0, .y = box.y0 }, + .{ .x = box.x0, .y = box.y1 }, + .{ .x = box.x1, .y = box.y0 }, + .{ .x = box.x1, .y = box.y1 }, + }; + const first = try mapPoint(matrix, corners[0]); + var result: BBox = .{ .x0 = first.x, .y0 = first.y, .x1 = first.x, .y1 = first.y }; + for (corners[1..]) |corner| { + const mapped = try mapPoint(matrix, corner); + result.x0 = @min(result.x0, mapped.x); + result.y0 = @min(result.y0, mapped.y); + result.x1 = @max(result.x1, mapped.x); + result.y1 = @max(result.y1, mapped.y); + } + return result; +} + +pub const BoxKind = enum { media, crop }; +pub const QuarterTurn = enum(u16) { + none = 0, + clockwise_90 = 90, + clockwise_180 = 180, + clockwise_270 = 270, +}; + +pub const PageView = struct { + pdf_bbox: BBox, + /// Physical points (1/72 inch), with the selected box's top-left at zero. + width_points: f64, + height_points: f64, + pdf_to_display: Matrix, + display_to_pdf: Matrix, +}; + +pub const PageGeometry = struct { + media_box: BBox, + /// CropBox intersected with MediaBox; defaults to MediaBox if absent. + crop_box: BBox, + rotation: QuarterTurn, + user_unit: f64, + media: PageView, + crop: PageView, + + pub fn init(media_box: BBox, crop_box: ?BBox, raw_rotate: i32, user_unit: f64) GeometryError!PageGeometry { + try validateBox(media_box, true); + if (!std.math.isFinite(user_unit) or user_unit <= 0 or user_unit > 75000) return error.InvalidUserUnit; + const normalized = @mod(raw_rotate, 360); + if (@mod(normalized, 90) != 0) return error.UnsupportedRotation; + const rotation: QuarterTurn = @fromBackingInt(@intCast(@as(u16, @intCast(normalized)))); + var effective_crop = media_box; + if (crop_box) |box| { + try validateBox(box, true); + effective_crop = intersect(media_box, box) orelse return error.EmptyCropBox; + } + return .{ + .media_box = media_box, + .crop_box = effective_crop, + .rotation = rotation, + .user_unit = user_unit, + .media = try makeView(media_box, rotation, user_unit), + .crop = try makeView(effective_crop, rotation, user_unit), + }; + } + + pub fn view(self: PageGeometry, kind: BoxKind) PageView { + return switch (kind) { + .media => self.media, + .crop => self.crop, + }; + } + + pub fn pdfToDisplay(self: PageGeometry, kind: BoxKind) Matrix { + return self.view(kind).pdf_to_display; + } + + pub fn displayToPdf(self: PageGeometry, kind: BoxKind) Matrix { + return self.view(kind).display_to_pdf; + } + + pub fn planCrop(self: PageGeometry, region: BBox, options: CropOptions) GeometryError!RasterPlan { + try validateBox(region, true); + if (!std.math.isFinite(options.dpi) or options.dpi <= 0) return error.InvalidDpi; + if (!std.math.isFinite(options.padding_points) or options.padding_points < 0) return error.InvalidPadding; + const selected = self.view(options.box); + const effective_dpi = options.dpi * self.user_unit; + if (!std.math.isFinite(effective_dpi) or effective_dpi <= 0) return error.InvalidDpi; + // Use the backend's effective DPI directly. Scaling physical extents by + // requested DPI separately can round an integer extent just above its + // grid edge while the crop matrix still maps that edge exactly. + const scale = effective_dpi / 72; + if (!std.math.isFinite(scale) or scale <= 0) return error.InvalidDpi; + const raster_view = try makeView(selected.pdf_bbox, self.rotation, 1); + // Poppler forms the canvas extent as (raw extent * effective DPI) / 72. + // Keep that evaluation order: 792 * (300 / 72) rounds above 3300. + const canvas_width = try pixelExtent(raster_view.width_points * effective_dpi / 72); + const canvas_height = try pixelExtent(raster_view.height_points * effective_dpi / 72); + const padding_pdf = options.padding_points / self.user_unit; + const padded: BBox = .{ + .x0 = region.x0 - padding_pdf, + .y0 = region.y0 - padding_pdf, + .x1 = region.x1 + padding_pdf, + .y1 = region.y1 + padding_pdf, + }; + try validateBox(padded, true); + const clamped = intersect(selected.pdf_bbox, padded) orelse return error.EmptyRegion; + const pdf_to_pixel = concat(.{ scale, 0, 0, scale, 0, 0 }, raster_view.pdf_to_display); + const pixels = try mapBBox(pdf_to_pixel, clamped); + const x = try pixelCoordinate(@max(0, @floor(pixels.x0)), canvas_width); + const y = try pixelCoordinate(@max(0, @floor(pixels.y0)), canvas_height); + const right = try pixelCoordinate(@min(@as(f64, @floatFromInt(canvas_width)), @ceil(pixels.x1)), canvas_width); + const bottom = try pixelCoordinate(@min(@as(f64, @floatFromInt(canvas_height)), @ceil(pixels.y1)), canvas_height); + if (right <= x or bottom <= y) return error.EmptyRegion; + const width = right - x; + const height = bottom - y; + const pixel_to_pdf = concat(try inverse(pdf_to_pixel), .{ 1, 0, 0, 1, @floatFromInt(x), @floatFromInt(y) }); + const actual = try mapBBox(pixel_to_pdf, .{ .x0 = 0, .y0 = 0, .x1 = @floatFromInt(width), .y1 = @floatFromInt(height) }); + return .{ + .box = options.box, + .requested_bbox = region, + .clamped_bbox = clamped, + .pdf_bbox = actual, + .x = x, + .y = y, + .width = width, + .height = height, + .canvas_width = canvas_width, + .canvas_height = canvas_height, + .pixel_to_pdf = pixel_to_pdf, + .requested_dpi = options.dpi, + .rasterizer_dpi = effective_dpi, + }; + } + + /// Full-page compatibility maps the complete accepted bitmap to exact PDF + /// extents, rather than expanding fractional page extents to a DPI grid. + pub fn fullPagePixelToPdf(self: PageGeometry, kind: BoxKind, accepted_width: u32, accepted_height: u32) GeometryError!Matrix { + if (accepted_width == 0 or accepted_height == 0 or accepted_width > max_raster_coordinate or accepted_height > max_raster_coordinate) return error.InvalidRasterDimensions; + const selected = self.view(kind); + const display_to_pixel: Matrix = .{ + @as(f64, @floatFromInt(accepted_width)) / selected.width_points, + 0, + 0, + @as(f64, @floatFromInt(accepted_height)) / selected.height_points, + 0, + 0, + }; + return inverse(concat(display_to_pixel, selected.pdf_to_display)); + } +}; + +pub const CropOptions = struct { + box: BoxKind = .media, + /// Physical points; divide by UserUnit before padding raw PDF coordinates. + padding_points: f64 = 0, + dpi: f64, +}; + +test "raster canvas preserves exact letter fallback dimensions at 300 DPI" { + const page = try PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 612, .y1 = 792 }, null, 0, 1); + const plan = try page.planCrop(page.media_box, .{ .dpi = 300 }); + try std.testing.expectEqual(@as(u32, 2550), plan.canvas_width); + try std.testing.expectEqual(@as(u32, 3300), plan.canvas_height); + try std.testing.expectEqual(plan.canvas_width, plan.width); + try std.testing.expectEqual(plan.canvas_height, plan.height); +} + +pub const RasterPlan = struct { + box: BoxKind, + requested_bbox: BBox, + clamped_bbox: BBox, + /// Outward DPI-grid extent; may exceed a fractional page edge by <1 pixel. + pdf_bbox: BBox, + x: u32, + y: u32, + width: u32, + height: u32, + canvas_width: u32, + canvas_height: u32, + /// Crop-local top-left pixel edges to canonical, unrotated PDF user space. + pixel_to_pdf: Matrix, + requested_dpi: f64, + /// The selected rasterizer ignores UserUnit, so physical scale uses this DPI. + rasterizer_dpi: f64, + + pub fn pixelCount(self: RasterPlan) u64 { + return @as(u64, self.width) * self.height; + } +}; + +const max_raster_coordinate: u32 = std.math.maxInt(i32); + +fn validateMatrix(matrix: Matrix) GeometryError!void { + for (matrix) |value| if (!std.math.isFinite(value)) return error.InvalidMatrix; +} + +fn validateBox(box: BBox, nonempty: bool) GeometryError!void { + if (!std.math.isFinite(box.x0) or !std.math.isFinite(box.y0) or !std.math.isFinite(box.x1) or !std.math.isFinite(box.y1)) return error.InvalidBox; + if (box.x1 < box.x0 or box.y1 < box.y0 or (nonempty and (box.x1 == box.x0 or box.y1 == box.y0))) return error.InvalidBox; + if (!std.math.isFinite(box.x1 - box.x0) or !std.math.isFinite(box.y1 - box.y0)) return error.InvalidBox; +} + +fn intersect(a: BBox, b: BBox) ?BBox { + const result: BBox = .{ .x0 = @max(a.x0, b.x0), .y0 = @max(a.y0, b.y0), .x1 = @min(a.x1, b.x1), .y1 = @min(a.y1, b.y1) }; + if (result.x0 >= result.x1 or result.y0 >= result.y1) return null; + return result; +} + +fn makeView(box: BBox, rotation: QuarterTurn, unit: f64) GeometryError!PageView { + const matrix: Matrix = switch (rotation) { + .none => .{ unit, 0, 0, -unit, -unit * box.x0, unit * box.y1 }, + .clockwise_90 => .{ 0, unit, unit, 0, -unit * box.y0, -unit * box.x0 }, + .clockwise_180 => .{ -unit, 0, 0, unit, unit * box.x1, -unit * box.y0 }, + .clockwise_270 => .{ 0, -unit, -unit, 0, unit * box.y1, unit * box.x1 }, + }; + const sideways = rotation == .clockwise_90 or rotation == .clockwise_270; + const width = (if (sideways) box.y1 - box.y0 else box.x1 - box.x0) * unit; + const height = (if (sideways) box.x1 - box.x0 else box.y1 - box.y0) * unit; + if (!std.math.isFinite(width) or !std.math.isFinite(height) or width <= 0 or height <= 0) return error.InvalidBox; + return .{ .pdf_bbox = box, .width_points = width, .height_points = height, .pdf_to_display = matrix, .display_to_pdf = try inverse(matrix) }; +} + +fn pixelExtent(value: f64) GeometryError!u32 { + if (!std.math.isFinite(value) or value <= 0) return error.RasterExtentOverflow; + const rounded = @ceil(value); + if (rounded > @as(f64, @floatFromInt(max_raster_coordinate))) return error.RasterExtentOverflow; + return @intFromFloat(rounded); +} + +fn pixelCoordinate(value: f64, limit: u32) GeometryError!u32 { + if (!std.math.isFinite(value) or value < 0 or value > @as(f64, @floatFromInt(limit))) return error.RasterExtentOverflow; + return @intFromFloat(value); +} + +fn expectMatrix(expected: Matrix, actual: Matrix) !void { + for (expected, actual) |e, a| try std.testing.expectApproxEqAbs(e, a, 1e-9); +} + +fn expectBox(expected: BBox, actual: BBox) !void { + try std.testing.expectApproxEqAbs(expected.x0, actual.x0, 1e-9); + try std.testing.expectApproxEqAbs(expected.y0, actual.y0, 1e-9); + try std.testing.expectApproxEqAbs(expected.x1, actual.x1, 1e-9); + try std.testing.expectApproxEqAbs(expected.y1, actual.y1, 1e-9); +} + +fn expectWindow(expected: [4]u32, actual: RasterPlan) !void { + try std.testing.expectEqual(expected[0], actual.x); + try std.testing.expectEqual(expected[1], actual.y); + try std.testing.expectEqual(expected[2], actual.width); + try std.testing.expectEqual(expected[3], actual.height); +} + +test "geometry affine composition inverse and four-corner bounds" { + const form: Matrix = .{ 0, 2, -2, 0, 80, 40 }; + const local: BBox = .{ .x0 = 0, .y0 = 0, .x1 = 10, .y1 = 5 }; + try expectBox(.{ .x0 = 70, .y0 = 40, .x1 = 80, .y1 = 60 }, try mapBBox(form, local)); + try expectMatrix(identity, concat(try inverse(form), form)); + try expectMatrix(.{ 0, 4, -6, 0, 240, 90 }, concat(.{ 3, 0, 0, 2, 0, 10 }, form)); + const diagonal: Matrix = .{ 1, 1, -1, 1, 0, 0 }; + try expectBox(.{ .x0 = -5, .y0 = 0, .x1 = 10, .y1 = 15 }, try mapBBox(diagonal, local)); + try std.testing.expectError(error.SingularMatrix, inverse(.{ 1, 2, 2, 4, 0, 0 })); + try std.testing.expectError(error.InvalidMatrix, inverse(.{ std.math.inf(f64), 0, 0, 1, 0, 0 })); + try std.testing.expectError(error.InvalidPoint, mapPoint(identity, .{ .x = std.math.nan(f64), .y = 0 })); +} + +test "geometry literal quarter-turn display and raster goldens at nonzero origin" { + const media: BBox = .{ .x0 = 10, .y0 = 20, .x1 = 110, .y1 = 220 }; + const region: BBox = .{ .x0 = 25, .y0 = 40, .x1 = 55, .y1 = 100 }; + const rotations = [_]i32{ 0, 90, 180, 270 }; + const expected_matrices = [_]Matrix{ + .{ 1, 0, 0, -1, -10, 220 }, + .{ 0, 1, 1, 0, -20, -10 }, + .{ -1, 0, 0, 1, 110, -20 }, + .{ 0, -1, -1, 0, 220, 110 }, + }; + const expected_display = [_]BBox{ + .{ .x0 = 15, .y0 = 120, .x1 = 45, .y1 = 180 }, + .{ .x0 = 20, .y0 = 15, .x1 = 80, .y1 = 45 }, + .{ .x0 = 55, .y0 = 20, .x1 = 85, .y1 = 80 }, + .{ .x0 = 120, .y0 = 55, .x1 = 180, .y1 = 85 }, + }; + const expected_windows = [_][4]u32{ .{ 30, 240, 60, 120 }, .{ 40, 30, 120, 60 }, .{ 110, 40, 60, 120 }, .{ 240, 110, 120, 60 } }; + const expected_canvas = [_][2]u32{ .{ 200, 400 }, .{ 400, 200 }, .{ 200, 400 }, .{ 400, 200 } }; + const expected_pixel_to_pdf = [_]Matrix{ + .{ 0.5, 0, 0, -0.5, 25, 100 }, + .{ 0, 0.5, 0.5, 0, 25, 40 }, + .{ -0.5, 0, 0, 0.5, 55, 40 }, + .{ 0, -0.5, -0.5, 0, 55, 100 }, + }; + for (rotations, 0..) |rotation, index| { + const page = try PageGeometry.init(media, null, rotation, 1); + try expectMatrix(expected_matrices[index], page.pdfToDisplay(.media)); + try expectBox(expected_display[index], try mapBBox(page.pdfToDisplay(.media), region)); + const plan = try page.planCrop(region, .{ .dpi = 144 }); + try expectWindow(expected_windows[index], plan); + try std.testing.expectEqual(expected_canvas[index][0], plan.canvas_width); + try std.testing.expectEqual(expected_canvas[index][1], plan.canvas_height); + try expectMatrix(expected_pixel_to_pdf[index], plan.pixel_to_pdf); + try expectBox(region, plan.pdf_bbox); + try expectBox(region, try mapBBox(page.displayToPdf(.media), expected_display[index])); + } +} + +test "geometry UserUnit physical scaling and literal Form coordinates are applied once" { + const media: BBox = .{ .x0 = 10, .y0 = 20, .x1 = 110, .y1 = 220 }; + const page = try PageGeometry.init(media, null, 90, 2); + const form: Matrix = .{ 0, 2, -2, 0, 80, 40 }; + const canonical = try mapBBox(form, .{ .x0 = 0, .y0 = 0, .x1 = 10, .y1 = 5 }); + try expectBox(.{ .x0 = 70, .y0 = 40, .x1 = 80, .y1 = 60 }, canonical); + try expectBox(.{ .x0 = 40, .y0 = 120, .x1 = 80, .y1 = 140 }, try mapBBox(page.pdfToDisplay(.media), canonical)); + const form_plan = try page.planCrop(canonical, .{ .dpi = 144 }); + try expectWindow(.{ 80, 240, 80, 40 }, form_plan); + try expectMatrix(.{ 0, 0.25, 0.25, 0, 70, 40 }, form_plan.pixel_to_pdf); + try std.testing.expectEqual(@as(f64, 288), form_plan.rasterizer_dpi); + const plan = try page.planCrop(.{ .x0 = 25, .y0 = 40, .x1 = 55, .y1 = 100 }, .{ .dpi = 144 }); + try expectWindow(.{ 80, 60, 240, 120 }, plan); + try std.testing.expectEqual(@as(u32, 800), plan.canvas_width); + try std.testing.expectEqual(@as(u32, 400), plan.canvas_height); + try std.testing.expectEqual(@as(u64, 28800), plan.pixelCount()); +} + +test "geometry crop intersection and independent MediaBox CropBox display origins" { + const media: BBox = .{ .x0 = 10, .y0 = 20, .x1 = 210, .y1 = 120 }; + const crop: BBox = .{ .x0 = 25, .y0 = 40, .x1 = 175, .y1 = 100 }; + const page = try PageGeometry.init(media, crop, 90, 1); + try std.testing.expectEqual(@as(f64, 100), page.media.width_points); + try std.testing.expectEqual(@as(f64, 200), page.media.height_points); + try std.testing.expectEqual(@as(f64, 60), page.crop.width_points); + try std.testing.expectEqual(@as(f64, 150), page.crop.height_points); + const point: Point = .{ .x = 40, .y = 50 }; + try std.testing.expectEqual(Point{ .x = 30, .y = 30 }, try mapPoint(page.pdfToDisplay(.media), point)); + try std.testing.expectEqual(Point{ .x = 10, .y = 15 }, try mapPoint(page.pdfToDisplay(.crop), point)); + const clipped = try PageGeometry.init(media, .{ .x0 = 0, .y0 = 0, .x1 = 175, .y1 = 100 }, 0, 1); + try expectBox(.{ .x0 = 10, .y0 = 20, .x1 = 175, .y1 = 100 }, clipped.crop_box); + const crop_plan = try page.planCrop(media, .{ .box = .crop, .dpi = 72 }); + try expectWindow(.{ 0, 0, 60, 150 }, crop_plan); + try expectBox(crop, crop_plan.pdf_bbox); +} + +test "geometry fractional origins outward grid and full-page extent compatibility" { + const media: BBox = .{ .x0 = 10.25, .y0 = 20.5, .x1 = 110.25, .y1 = 220.5 }; + const region: BBox = .{ .x0 = 25.2, .y0 = 35.3, .x1 = 50.4, .y1 = 70.6 }; + for ([_]i32{ 0, 90 }, 0..) |rotation, index| { + const page = try PageGeometry.init(media, null, rotation, 1); + const plan = try page.planCrop(region, .{ .dpi = 144 }); + try expectWindow(([_][4]u32{ .{ 29, 299, 52, 72 }, .{ 29, 29, 72, 52 } })[index], plan); + try expectBox(.{ .x0 = 24.75, .y0 = 35, .x1 = 50.75, .y1 = 71 }, plan.pdf_bbox); + } + const fractional: BBox = .{ .x0 = 0, .y0 = 0, .x1 = 9.8, .y1 = 19.8 }; + const unrotated = try PageGeometry.init(fractional, null, 0, 1); + try expectMatrix(.{ 0.98, 0, 0, -0.99, 0, 19.8 }, try unrotated.fullPagePixelToPdf(.media, 10, 20)); + const rotated = try PageGeometry.init(fractional, null, 90, 2); + try expectMatrix(.{ 0, 0.99, 0.98, 0, 0, 0 }, try rotated.fullPagePixelToPdf(.media, 20, 10)); + // The quarter rotation swaps extents: PDF x spans the bitmap's height. + try expectBox(fractional, try mapBBox(try rotated.fullPagePixelToPdf(.media, 20, 10), .{ .x0 = 0, .y0 = 0, .x1 = 20, .y1 = 10 })); + const probe = try PageGeometry.init(.{ .x0 = 10, .y0 = 20, .x1 = 210, .y1 = 120 }, null, 0, 1); + const fractional_dpi = try probe.planCrop(probe.media_box, .{ .dpi = 144.5 }); + try std.testing.expectEqual(@as(u32, 402), fractional_dpi.canvas_width); + try std.testing.expectEqual(@as(u32, 201), fractional_dpi.canvas_height); + try std.testing.expectEqual(@as(f64, 144.5), fractional_dpi.rasterizer_dpi); +} + +test "geometry clamping physical padding and fractional edge expansion" { + const page = try PageGeometry.init(.{ .x0 = 10, .y0 = 20, .x1 = 110, .y1 = 220 }, null, 0, 2); + const plan = try page.planCrop(.{ .x0 = 12, .y0 = 40, .x1 = 20, .y1 = 50 }, .{ .padding_points = 10, .dpi = 72 }); + try expectBox(.{ .x0 = 10, .y0 = 35, .x1 = 25, .y1 = 55 }, plan.clamped_bbox); + try expectWindow(.{ 0, 330, 30, 40 }, plan); + try std.testing.expectError(error.EmptyRegion, page.planCrop(.{ .x0 = 200, .y0 = 300, .x1 = 210, .y1 = 310 }, .{ .dpi = 72 })); + const fractional = try PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 9.8, .y1 = 19.8 }, null, 0, 1); + const full_grid = try fractional.planCrop(fractional.media_box, .{ .dpi = 72 }); + try expectWindow(.{ 0, 0, 10, 20 }, full_grid); + try expectBox(.{ .x0 = 0, .y0 = -0.2, .x1 = 10, .y1 = 19.8 }, full_grid.pdf_bbox); +} + +test "geometry literal full-page extent matrices for every quarter and UserUnit" { + const box: BBox = .{ .x0 = 10, .y0 = 20, .x1 = 19.8, .y1 = 39.8 }; + const rotations = [_]i32{ 0, 90, 180, 270 }; + const dimensions = [_][2]u32{ .{ 10, 20 }, .{ 20, 10 }, .{ 10, 20 }, .{ 20, 10 } }; + const expected = [_]Matrix{ + .{ 0.98, 0, 0, -0.99, 10, 39.8 }, + .{ 0, 0.99, 0.98, 0, 10, 20 }, + .{ -0.98, 0, 0, 0.99, 19.8, 20 }, + .{ 0, -0.99, -0.98, 0, 19.8, 39.8 }, + }; + for ([_]f64{ 1, 2 }) |unit| { + for (rotations, 0..) |rotation, index| { + const page = try PageGeometry.init(box, null, rotation, unit); + const matrix = try page.fullPagePixelToPdf(.media, dimensions[index][0], dimensions[index][1]); + try expectMatrix(expected[index], matrix); + try expectBox(box, try mapBBox(matrix, .{ + .x0 = 0, + .y0 = 0, + .x1 = @floatFromInt(dimensions[index][0]), + .y1 = @floatFromInt(dimensions[index][1]), + })); + } + } +} + +test "geometry fractional UserUnit raster scale matches backend canvas seam" { + // Installed Poppler 26.02 renders MediaBox [0 0 .9 10], UserUnit 3, + // with effective -r 240 as 3x34. Physical display metadata remains 2.7x30. + const page = try PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 0.9, .y1 = 10 }, null, 0, 3); + const plan = try page.planCrop(page.media_box, .{ .dpi = 80 }); + try std.testing.expectApproxEqAbs(@as(f64, 2.7), page.media.width_points, 1e-12); + try std.testing.expectEqual(@as(f64, 240), plan.rasterizer_dpi); + try std.testing.expectEqual(@as(u32, 3), plan.canvas_width); + try std.testing.expectEqual(@as(u32, 34), plan.canvas_height); + try expectWindow(.{ 0, 0, 3, 34 }, plan); + try expectMatrix(.{ 0.3, 0, 0, -0.3, 0, 10 }, plan.pixel_to_pdf); +} + +test "geometry rotation normalization finite inputs and bounded raster integers" { + const box: BBox = .{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 200 }; + try std.testing.expectEqual(QuarterTurn.clockwise_270, (try PageGeometry.init(box, null, -90, 1)).rotation); + try std.testing.expectEqual(QuarterTurn.clockwise_90, (try PageGeometry.init(box, null, 450, 1)).rotation); + try std.testing.expectError(error.UnsupportedRotation, PageGeometry.init(box, null, 45, 1)); + for ([_]f64{ 0, -1, 75000.1, std.math.nan(f64), std.math.inf(f64) }) |unit| { + try std.testing.expectError(error.InvalidUserUnit, PageGeometry.init(box, null, 0, unit)); + } + _ = try PageGeometry.init(box, null, 0, 75000); + try std.testing.expectError(error.InvalidBox, PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 0, .y1 = 1 }, null, 0, 1)); + try std.testing.expectError(error.InvalidBox, PageGeometry.init(.{ .x0 = 2, .y0 = 0, .x1 = 1, .y1 = 1 }, null, 0, 1)); + try std.testing.expectError(error.InvalidBox, PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = std.math.nan(f64), .y1 = 1 }, null, 0, 1)); + try std.testing.expectError(error.EmptyCropBox, PageGeometry.init(box, .{ .x0 = 300, .y0 = 0, .x1 = 400, .y1 = 100 }, 0, 1)); + const page = try PageGeometry.init(box, null, 0, 1); + for ([_]f64{ 0, -1, std.math.nan(f64), std.math.inf(f64) }) |dpi| { + try std.testing.expectError(error.InvalidDpi, page.planCrop(box, .{ .dpi = dpi })); + } + try std.testing.expectError(error.InvalidPadding, page.planCrop(box, .{ .dpi = 72, .padding_points = -1 })); + try std.testing.expectError(error.InvalidRasterDimensions, page.fullPagePixelToPdf(.media, 0, 1)); + try std.testing.expectError(error.InvalidRasterDimensions, page.fullPagePixelToPdf(.media, std.math.maxInt(u32), 1)); + const huge = try PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 2147483648, .y1 = 1 }, null, 0, 1); + try std.testing.expectError(error.RasterExtentOverflow, huge.planCrop(huge.media_box, .{ .dpi = 72 })); + const edge = try PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 2147483647, .y1 = 1 }, null, 0, 1); + const bounded = try edge.planCrop(edge.media_box, .{ .dpi = 72 }); + try std.testing.expectEqual(max_raster_coordinate, bounded.width); + try std.testing.expectEqual(@as(u64, max_raster_coordinate), bounded.pixelCount()); +} diff --git a/src/pagetree.zig b/src/pagetree.zig index 117aeb8..a09b8e6 100644 --- a/src/pagetree.zig +++ b/src/pagetree.zig @@ -27,6 +27,8 @@ pub const Page = struct { crop_box: [4]f64, /// Rotation in degrees (0, 90, 180, 270) rotation: i32, + /// Leaf-only physical scale per PDF user-space unit; NaN if specified but invalid. + user_unit: f64 = 1, /// Inherited Resources dictionary resources: ?Object.Dict, }; @@ -526,18 +528,46 @@ fn walkPageTree( } } else if (std.mem.eql(u8, type_name, "Page")) { // Leaf node - add to pages list + const user_unit = try leafUserUnit(allocator, data, xref, dict, cache, options); pages.append(allocator, .{ .ref = node_ref, .dict = dict, .media_box = mediabox, .crop_box = cropbox orelse mediabox, .rotation = rotation, + .user_unit = user_unit, .resources = resources, }) catch return PageTreeError.OutOfMemory; } // Ignore unknown types } +fn leafUserUnit( + allocator: std.mem.Allocator, + data: []const u8, + xref: *const XRefTable, + dict: Object.Dict, + cache: *std.AutoHashMap(u32, Object), + options: BuildOptions, +) PageTreeError!f64 { + var value = dict.get("UserUnit") orelse return 1; + // UserUnit is not inherited. Bound indirect chains, including cyclic refs, + // and retain invalid specified values as unavailable geometry, not unit 1. + var remaining: usize = 16; + while (value == .reference) { + if (remaining == 0) return std.math.nan(f64); + remaining -= 1; + value = resolveRefWithOptions(allocator, data, xref, value.reference, cache, options) catch return error.OutOfMemory; + } + const number: f64 = switch (value) { + .integer => |integer| @floatFromInt(integer), + .real => |real| real, + else => return std.math.nan(f64), + }; + if (!std.math.isFinite(number) or number <= 0 or number > 75000) return std.math.nan(f64); + return number; +} + fn emit(allocator: std.mem.Allocator, options: BuildOptions, diagnostic: structural.Diagnostic) void { structural.appendDiagnostic(options.diagnostic_allocator orelse allocator, options.diagnostics, diagnostic); } @@ -676,3 +706,82 @@ test "extractBox" { _ = allocator; } + +fn userUnitPageFixture(allocator: std.mem.Allocator, data: []const u8, object_count: u32) ![]Page { + var xref = XRefTable.init(allocator); + defer xref.deinit(); + var trailer_entries = [_]Object.Dict.Entry{.{ .key = "Root", .value = .{ .reference = .{ .num = 1, .gen = 0 } } }}; + xref.trailer = .{ .entries = &trailer_entries }; + var number: u32 = 1; + while (number <= object_count) : (number += 1) { + var needle_buffer: [32]u8 = undefined; + const needle = try std.fmt.bufPrint(&needle_buffer, "{d} 0 obj", .{number}); + const offset = std.mem.indexOf(u8, data, needle) orelse return error.MissingTestObject; + try xref.entries.put(number, .{ .offset = offset, .gen_or_index = 0, .entry_type = .in_use }); + } + return buildPageTreeWithOptions(allocator, data, &xref, .{ .recover = false }); +} + +test "UserUnit is leaf only while page boxes and rotation remain inherited" { + var arena = std.heap.ArenaAllocator.init(std.testing.allocator); + defer arena.deinit(); + const data = + "1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n" ++ + "2 0 obj << /Type /Pages /Count 2 /Kids [3 0 R 4 0 R] /UserUnit 3 /MediaBox [10 20 110 220] /CropBox [25 40 100 200] /Rotate 90 >> endobj\n" ++ + "3 0 obj << /Type /Page /Parent 2 0 R /UserUnit 2 >> endobj\n" ++ + "4 0 obj << /Type /Page /Parent 2 0 R >> endobj\n"; + const pages = try userUnitPageFixture(arena.allocator(), data, 4); + try std.testing.expectEqual(@as(usize, 2), pages.len); + try std.testing.expectEqual(@as(f64, 2), pages[0].user_unit); + try std.testing.expectEqual(@as(f64, 1), pages[1].user_unit); + for (pages) |page| { + try std.testing.expectEqualSlices(f64, &.{ 10, 20, 110, 220 }, &page.media_box); + try std.testing.expectEqualSlices(f64, &.{ 25, 40, 100, 200 }, &page.crop_box); + try std.testing.expectEqual(@as(i32, 90), page.rotation); + } +} + +test "UserUnit resolves indirect integer and real numbers without fabricating invalid values" { + var arena = std.heap.ArenaAllocator.init(std.testing.allocator); + defer arena.deinit(); + const data = + "1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n" ++ + "2 0 obj << /Type /Pages /Count 4 /Kids [3 0 R 4 0 R 5 0 R 6 0 R] /MediaBox [0 0 100 200] >> endobj\n" ++ + "3 0 obj << /Type /Page /Parent 2 0 R /UserUnit 7 0 R >> endobj\n" ++ + "4 0 obj << /Type /Page /Parent 2 0 R /UserUnit 8 0 R >> endobj\n" ++ + "5 0 obj << /Type /Page /Parent 2 0 R /UserUnit 0 >> endobj\n" ++ + "6 0 obj << /Type /Page /Parent 2 0 R /UserUnit /Invalid >> endobj\n" ++ + "7 0 obj 1.5 endobj\n" ++ + "8 0 obj 2 endobj\n"; + const pages = try userUnitPageFixture(arena.allocator(), data, 8); + try std.testing.expectEqual(@as(usize, 4), pages.len); + try std.testing.expectEqual(@as(f64, 1.5), pages[0].user_unit); + try std.testing.expectEqual(@as(f64, 2), pages[1].user_unit); + try std.testing.expect(std.math.isNan(pages[2].user_unit)); + try std.testing.expect(std.math.isNan(pages[3].user_unit)); +} + +test "UserUnit invalid numbers missing refs and cyclic refs stay unavailable" { + var xref = XRefTable.init(std.testing.allocator); + defer xref.deinit(); + var cache = std.AutoHashMap(u32, Object).init(std.testing.allocator); + defer cache.deinit(); + try cache.put(7, .{ .reference = .{ .num = 8, .gen = 0 } }); + try cache.put(8, .{ .reference = .{ .num = 7, .gen = 0 } }); + var entries = [_]Object.Dict.Entry{.{ .key = "UserUnit", .value = .{ .integer = 0 } }}; + const dict: Object.Dict = .{ .entries = &entries }; + const values = [_]Object{ + .{ .integer = -1 }, + .{ .integer = 75001 }, + .{ .real = std.math.nan(f64) }, + .{ .real = std.math.inf(f64) }, + .{ .reference = .{ .num = 99, .gen = 0 } }, + .{ .reference = .{ .num = 7, .gen = 0 } }, + }; + for (values) |value| { + entries[0].value = value; + try std.testing.expect(std.math.isNan(try leafUserUnit(std.testing.allocator, "", &xref, dict, &cache, .{}))); + } + entries[0].value = .{ .integer = 75000 }; + try std.testing.expectEqual(@as(f64, 75000), try leafUserUnit(std.testing.allocator, "", &xref, dict, &cache, .{})); +} diff --git a/src/paint.zig b/src/paint.zig index 7f8282a..0cf6649 100644 --- a/src/paint.zig +++ b/src/paint.zig @@ -1,19 +1,10 @@ //! Conservative content-stream graphics evidence in PDF user space. const std = @import("std"); const layout = @import("layout.zig"); +const page_geometry = @import("page_geometry.zig"); -pub const identity: [6]f64 = .{ 1, 0, 0, 1, 0, 0 }; - -pub fn concat(outer: [6]f64, inner: [6]f64) [6]f64 { - return .{ - outer[0] * inner[0] + outer[2] * inner[1], - outer[1] * inner[0] + outer[3] * inner[1], - outer[0] * inner[2] + outer[2] * inner[3], - outer[1] * inner[2] + outer[3] * inner[3], - outer[0] * inner[4] + outer[2] * inner[5] + outer[4], - outer[1] * inner[4] + outer[3] * inner[5] + outer[5], - }; -} +pub const identity = page_geometry.identity; +pub const concat = page_geometry.concat; pub const State = struct { ctm: [6]f64 = identity, diff --git a/src/reconcile.zig b/src/reconcile.zig index bd57019..cc9e288 100644 --- a/src/reconcile.zig +++ b/src/reconcile.zig @@ -795,7 +795,7 @@ fn shouldStartBlock(previous: ReconciledSpan, current: ReconciledSpan, options: switch (boundary) { .join, .space => return false, .block_break, .region_break => return true, - .line_break => {}, + .line_break => if (current.span.source == .fresh_ocr) return false, } } if (kindForSpan(previous.span) != kindForSpan(current.span)) return true; @@ -809,7 +809,7 @@ fn shouldStartBlock(previous: ReconciledSpan, current: ReconciledSpan, options: fn nativeBoundary(previous: TextSpan, current: TextSpan, options: ReconcileOptions) ?layout.TextBoundary { if (!options.preserve_input_order or previous.page_index != current.page_index or previous.source != current.source or - (current.source != .native_pdf and current.source != .embedded_ocr)) return null; + (current.source != .native_pdf and current.source != .embedded_ocr and current.source != .fresh_ocr)) return null; const boundary = current.leading_boundary orelse return null; if ((boundary == .join or boundary == .space) and (previous.line_id == null or current.line_id == null or previous.line_id.? != current.line_id.?)) return null; @@ -1341,6 +1341,56 @@ test "native classified joins preserve superscripts and classified line breaks" try std.testing.expectEqualStrings("F1.25\nNext", doc.chunks[0].content); } +test "fresh OCR explicit boundaries survive inverse quarter-turn PDF boxes" { + const geometry = @import("page_geometry.zig"); + const word_boxes = [_]BBox{ + .{ .x0 = 10, .y0 = 10, .x1 = 30, .y1 = 20 }, + .{ .x0 = 40, .y0 = 10, .x1 = 60, .y1 = 20 }, + .{ .x0 = 10, .y0 = 30, .x1 = 40, .y1 = 40 }, + }; + for ([_]i32{ 90, 180, 270 }) |rotation| { + const page = try geometry.PageGeometry.init(.{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 200 }, null, rotation, 1); + var spans: [3]TextSpan = undefined; + for (word_boxes, 0..) |box, index| { + spans[index] = TextSpan.init(.{ + .bbox = try geometry.mapBBox(page.displayToPdf(.media), box), + .text = ([_][]const u8{ "ALPHA", "BETA", "GAMMA" })[index], + .source = .fresh_ocr, + .font = .{ .size = 10 }, + .line_id = if (index < 2) 0 else 1, + .leading_boundary = ([_]layout.TextBoundary{ .region_break, .space, .line_break })[index], + }); + } + var doc = try reconcile(std.testing.allocator, &.{.{ .spans = &spans }}, .{ .preserve_input_order = true }); + defer doc.deinit(); + try std.testing.expectEqual(@as(usize, 1), doc.blocks.len); + try std.testing.expectEqualStrings("ALPHA BETA\nGAMMA", doc.blocks[0].text); + try std.testing.expectEqualStrings("ALPHA BETA\nGAMMA", doc.chunks[0].content); + try std.testing.expectEqual(@as(usize, 6), doc.spans[1].block_text_start); + try std.testing.expectEqual(@as(usize, 11), doc.spans[2].block_text_start); + } +} + +test "fresh OCR independent crop boundaries separate overlapping PDF regions" { + var first = testSpan("FIRST", .fresh_ocr, 10, 10, 20, 30); + first.line_id = 0; + first.leading_boundary = .region_break; + var second = testSpan("SECOND", .fresh_ocr, 10, 10, 20, 30); + second.line_id = 0; + second.leading_boundary = .region_break; + var third = testSpan("THIRD", .fresh_ocr, 10, 10, 20, 30); + third.line_id = 0; + third.leading_boundary = .block_break; + var doc = try reconcile(std.testing.allocator, &.{.{ .spans = &.{ first, second, third } }}, .{ .preserve_input_order = true }); + defer doc.deinit(); + try std.testing.expectEqual(@as(usize, 3), doc.blocks.len); + try std.testing.expectEqualStrings("FIRST", doc.blocks[0].text); + try std.testing.expectEqualStrings("SECOND", doc.blocks[1].text); + try std.testing.expectEqualStrings("THIRD", doc.blocks[2].text); + try std.testing.expectEqualStrings("FIRST\n\nSECOND\n\nTHIRD", doc.chunks[0].content); + for (doc.spans) |span| try std.testing.expectEqual(@as(usize, 0), span.block_text_start); +} + test "coordinate serializers include hocr alto and debug overlay" { const native = [_]TextSpan{ testSpan("Hello ", .native_pdf, 10, 700, 80, 712), diff --git a/src/root.zig b/src/root.zig index dc98811..136e2d9 100644 --- a/src/root.zig +++ b/src/root.zig @@ -29,6 +29,8 @@ pub const encryption = @import("encryption.zig"); pub const structural = @import("structural.zig"); pub const simd = @import("simd.zig"); pub const layout = @import("layout.zig"); +pub const page_geometry = @import("page_geometry.zig"); +pub const vector_evidence = @import("vector_evidence.zig"); pub const native_layout = @import("native_layout.zig"); pub const text_search = @import("text_search.zig"); pub const structtree = @import("structtree.zig"); @@ -1839,6 +1841,19 @@ pub const Document = struct { rotation: i32, }; + /// Canonical PDF coordinates remain unchanged. These page-local matrices + /// map them to physical points at the selected box's top-left display origin. + pub fn getPageGeometry(self: *const Document, page_num: usize) ?page_geometry.PageGeometry { + if (page_num >= self.pages.items.len) return null; + const page = self.pages.items[page_num]; + return page_geometry.PageGeometry.init( + .{ .x0 = page.media_box[0], .y0 = page.media_box[1], .x1 = page.media_box[2], .y1 = page.media_box[3] }, + .{ .x0 = page.crop_box[0], .y0 = page.crop_box[1], .x1 = page.crop_box[2], .y1 = page.crop_box[3] }, + page.rotation, + page.user_unit, + ) catch null; + } + // ========================================================================= // Feature 1: Document Metadata (/Info dict) // ========================================================================= @@ -2291,9 +2306,9 @@ pub const Document = struct { }); break :blk null; }; - if (geometry) |*page_geometry| { - defer page_geometry.deinit(); - const canonical_glyphs = page_geometry.semantic_glyphs orelse page_geometry.glyphs; + if (geometry) |*native_geometry| { + defer native_geometry.deinit(); + const canonical_glyphs = native_geometry.semantic_glyphs orelse native_geometry.glyphs; const raw_or_empty: []const u8 = if (raw) |raw_text| raw_text else ""; const page = self.pages.items[page_idx]; var native_result = native_layout.analyze( @@ -3017,8 +3032,8 @@ pub const Document = struct { /// Rasterize one routed formula region to an ephemeral PNG. The crop is /// page-clamped and expressed in Poppler's top-left pixel coordinates. - /// Rotated pages remain explicit unsupported outcomes until their geometry - /// mapping is fixture-gated. + /// Quarter-turn pages use the shared page transform; the legacy requested + /// bbox is retained alongside the crop-local pixel-to-PDF mapping. pub fn rasterizeFormulaRegionDetailed( self: *Document, allocator: std.mem.Allocator, @@ -3032,26 +3047,23 @@ pub const Document = struct { .diagnostic_code = "formula_crop_requires_file", }; const page = self.pages.items[page_idx]; - const normalized_rotation = @mod(page.rotation, 360); - if (normalized_rotation != 0) return .{ + const page_model = self.getPageGeometry(page_idx) orelse return .{ .status = .unavailable, - .diagnostic_code = "formula_crop_rotation_unsupported", - }; - - const page_bbox = layout.BBox{ - .x0 = page.media_box[0], - .y0 = page.media_box[1], - .x1 = page.media_box[2], - .y1 = page.media_box[3], + .diagnostic_code = if (@mod(page.rotation, 90) != 0) "formula_crop_rotation_unsupported" else "invalid_formula_crop_geometry", }; - const geometry = formulaCropGeometry(page_bbox, bbox, config.crop_padding_points, config.crop_dpi) orelse return .{ + const geometry = page_model.planCrop(bbox, .{ .padding_points = config.crop_padding_points, .dpi = @floatFromInt(config.crop_dpi) }) catch return .{ .status = .invalid_output, .diagnostic_code = "invalid_formula_crop_geometry", }; + const raster_limits = ocr.OcrConfig{}; + if (!@import("ocr/crop.zig").withinPixelLimit(geometry.width, geometry.height, raster_limits.max_image_pixels)) return .{ + .status = .invalid_output, + .diagnostic_code = "formula_crop_pixel_limit", + }; const page_number = try std.fmt.allocPrint(allocator, "{}", .{page_idx + 1}); defer allocator.free(page_number); - const dpi_arg = try std.fmt.allocPrint(allocator, "{}", .{config.crop_dpi}); + const dpi_arg = try std.fmt.allocPrint(allocator, "{d}", .{geometry.rasterizer_dpi}); defer allocator.free(dpi_arg); const x_arg = try std.fmt.allocPrint(allocator, "{}", .{geometry.x}); defer allocator.free(x_arg); @@ -3126,7 +3138,7 @@ pub const Document = struct { .diagnostic_code = "formula_rasterizer_failed", }, } - const dimensions = readPngDimensions(allocator, image_path) catch |err| switch (err) { + const dimensions = @import("ocr/crop.zig").readIdentity(allocator, image_path, raster_limits.max_image_bytes, raster_limits.max_image_pixels) catch |err| switch (err) { error.OutOfMemory => return err, else => return .{ .status = .invalid_output, @@ -3135,12 +3147,20 @@ pub const Document = struct { .diagnostic_code = "invalid_formula_crop_image", }, }; + if (dimensions.width != geometry.width or dimensions.height != geometry.height) return .{ + .status = .invalid_output, + .stderr = result.stderr, + .duration_ms = elapsedMilliseconds(started_ns), + .diagnostic_code = "invalid_formula_crop_image", + }; keep_image_path = true; return .{ .status = .completed, .input = .{ .image_path = image_path, - .pdf_bbox = geometry.pdf_bbox, + .pdf_bbox = geometry.clamped_bbox, + .pixel_to_pdf = geometry.pixel_to_pdf, + .rasterizer_dpi = geometry.rasterizer_dpi, .pixel_width = dimensions.width, .pixel_height = dimensions.height, .dpi = config.crop_dpi, @@ -3159,31 +3179,14 @@ pub const Document = struct { }; fn formulaCropGeometry(page_bbox: layout.BBox, bbox: layout.BBox, padding: f64, dpi: u32) ?FormulaCropGeometry { - const crop_bbox = layout.BBox{ - .x0 = @max(page_bbox.x0, bbox.x0 - padding), - .y0 = @max(page_bbox.y0, bbox.y0 - padding), - .x1 = @min(page_bbox.x1, bbox.x1 + padding), - .y1 = @min(page_bbox.y1, bbox.y1 + padding), - }; - if (!(std.math.isFinite(page_bbox.x0) and std.math.isFinite(page_bbox.y0) and - std.math.isFinite(page_bbox.x1) and std.math.isFinite(page_bbox.y1) and - std.math.isFinite(crop_bbox.x0) and std.math.isFinite(crop_bbox.y0) and - std.math.isFinite(crop_bbox.x1) and std.math.isFinite(crop_bbox.y1)) or - page_bbox.x1 <= page_bbox.x0 or page_bbox.y1 <= page_bbox.y0 or - crop_bbox.x1 <= crop_bbox.x0 or crop_bbox.y1 <= crop_bbox.y0) return null; - const scale = @as(f64, @floatFromInt(dpi)) / 72.0; - const left = @floor((crop_bbox.x0 - page_bbox.x0) * scale); - const top = @floor((page_bbox.y1 - crop_bbox.y1) * scale); - const right = @ceil((crop_bbox.x1 - page_bbox.x0) * scale); - const bottom = @ceil((page_bbox.y1 - crop_bbox.y0) * scale); - const max_u32_f = @as(f64, @floatFromInt(std.math.maxInt(u32))); - if (left < 0 or top < 0 or right <= left or bottom <= top or right > max_u32_f or bottom > max_u32_f) return null; + const model = page_geometry.PageGeometry.init(page_bbox, page_bbox, 0, 1) catch return null; + const plan = model.planCrop(bbox, .{ .padding_points = padding, .dpi = @floatFromInt(dpi) }) catch return null; return .{ - .pdf_bbox = crop_bbox, - .x = @intFromFloat(left), - .y = @intFromFloat(top), - .width = @intFromFloat(right - left), - .height = @intFromFloat(bottom - top), + .pdf_bbox = plan.clamped_bbox, + .x = plan.x, + .y = plan.y, + .width = plan.width, + .height = plan.height, }; } @@ -3215,8 +3218,8 @@ pub const Document = struct { } /// Caller owns the ephemeral image path and must delete and free it after - /// recognition. Rotated pages are unsupported until coordinate mapping is - /// fixture-proven. Limits bound accepted raster pixels/bytes, not process RSS. + /// recognition. Quarter-turn mapping uses validated page geometry. Limits + /// bound accepted raster pixels/bytes, not process RSS. pub fn rasterizeOcrRegionDetailed(self: *Document, allocator: std.mem.Allocator, page_idx: usize, bbox: layout.BBox, config: ocr.OcrConfig) !ocr.RasterizeOutcome { return self.rasterizeOcrDetailed(allocator, page_idx, bbox, config, true); } @@ -3229,17 +3232,23 @@ pub const Document = struct { }; const page = self.pages.items[page_idx]; - if (@mod(page.rotation, 360) != 0) return .{ .status = .unavailable, .diagnostic_code = .rasterizer_rotation_unsupported }; const crop = @import("ocr/crop.zig"); const page_bbox = layout.BBox{ .x0 = page.media_box[0], .y0 = page.media_box[1], .x1 = page.media_box[2], .y1 = page.media_box[3] }; - const geometry = crop.geometry(page_bbox, bbox, if (regional) config.crop_padding_points else 0, config.dpi) orelse return .{ .status = .invalid_output, .diagnostic_code = .invalid_crop_geometry }; + const page_model = self.getPageGeometry(page_idx) orelse return .{ + .status = .unavailable, + .diagnostic_code = if (@mod(page.rotation, 90) != 0) .rasterizer_rotation_unsupported else .invalid_crop_geometry, + }; + const geometry = page_model.planCrop(bbox, .{ .padding_points = if (regional) config.crop_padding_points else 0, .dpi = @floatFromInt(config.dpi) }) catch |err| return .{ + .status = .invalid_output, + .diagnostic_code = if (err == error.RasterExtentOverflow) .rasterizer_pixel_limit else .invalid_crop_geometry, + }; if (!crop.withinPixelLimit(geometry.width, geometry.height, config.max_image_pixels)) return .{ .status = .invalid_output, .diagnostic_code = .rasterizer_pixel_limit }; if (config.max_image_bytes < 24) return .{ .status = .invalid_output, .diagnostic_code = .rasterizer_image_byte_limit }; const page_number = try std.fmt.allocPrint(allocator, "{}", .{page_idx + 1}); defer allocator.free(page_number); - const dpi_arg = try std.fmt.allocPrint(allocator, "{}", .{config.dpi}); + const dpi_arg = try std.fmt.allocPrint(allocator, "{d}", .{geometry.rasterizer_dpi}); defer allocator.free(dpi_arg); const x_arg = try std.fmt.allocPrint(allocator, "{}", .{geometry.x}); defer allocator.free(x_arg); @@ -3367,7 +3376,13 @@ pub const Document = struct { }, }, }; - if (regional and (dimensions.width != geometry.width or dimensions.height != geometry.height)) return .{ .status = .invalid_output, .stderr = result.stderr, .duration_ms = elapsedMilliseconds(started_ns), .diagnostic_code = .invalid_raster_image }; + if (dimensions.width != geometry.width or dimensions.height != geometry.height) return .{ .status = .invalid_output, .stderr = result.stderr, .duration_ms = elapsedMilliseconds(started_ns), .diagnostic_code = .invalid_raster_image }; + const pixel_to_pdf = if (regional) geometry.pixel_to_pdf else page_model.fullPagePixelToPdf(.media, dimensions.width, dimensions.height) catch return .{ + .status = .invalid_output, + .stderr = result.stderr, + .duration_ms = elapsedMilliseconds(started_ns), + .diagnostic_code = .invalid_raster_image, + }; keep_image_path = true; return .{ .status = .completed, @@ -3376,6 +3391,8 @@ pub const Document = struct { // Preserve the established full-page OCR mapping. Regional // crops map through their actual outward pixel boundaries. .pdf_bbox = if (regional) geometry.pdf_bbox else page_bbox, + .pixel_to_pdf = pixel_to_pdf, + .rasterizer_dpi = geometry.rasterizer_dpi, .image_path = image_path, .pixel_width = dimensions.width, .pixel_height = dimensions.height, @@ -3392,20 +3409,6 @@ pub const Document = struct { return if (elapsed_ns > 0) @intCast(@divTrunc(elapsed_ns, 1_000_000)) else 0; } - fn readPngDimensions(allocator: std.mem.Allocator, image_path: []const u8) !struct { width: u32, height: u32 } { - const data = try runtime.readFileAllocAlignedCwd(allocator, image_path, .fromByteUnits(1)); - defer allocator.free(data); - - const png_signature = "\x89PNG\r\n\x1a\n"; - if (data.len < 24 or !std.mem.eql(u8, data[0..8], png_signature)) return error.InvalidRasterImage; - if (!std.mem.eql(u8, data[12..16], "IHDR")) return error.InvalidRasterImage; - - const width = std.mem.readInt(u32, data[16..][0..4], .big); - const height = std.mem.readInt(u32, data[20..][0..4], .big); - if (width == 0 or height == 0) return error.InvalidRasterImage; - return .{ .width = width, .height = height }; - } - // ========================================================================= // Feature 6: Image Detection // ========================================================================= @@ -3430,6 +3433,110 @@ pub const Document = struct { } /// Detect images from already-decoded page content. + /// Painted vector-path inventory, independent of text and image extraction. + pub fn getPageVectorEvidenceFromContent(self: *Document, page_idx: usize, allocator: std.mem.Allocator, scratch_allocator: std.mem.Allocator, content: []const u8) !vector_evidence.Evidence { + return self.getPageVectorEvidenceFromContentWithLimits(page_idx, allocator, scratch_allocator, content, .{}); + } + + pub fn getPageVectorEvidenceFromContentWithLimits(self: *Document, page_idx: usize, allocator: std.mem.Allocator, scratch_allocator: std.mem.Allocator, content: []const u8, limits: vector_evidence.Limits) !vector_evidence.Evidence { + if (page_idx >= self.pages.items.len) return error.PageNotFound; + var builder = vector_evidence.Builder{ .allocator = allocator, .limits = limits }; + defer builder.deinit(); + var traversal = form_scope.Traversal{ .page_index = @intCast(page_idx), .max_invocations = limits.max_form_invocations }; + const context = self.contentContext(page_idx, scratch_allocator, &traversal); + try self.collectVectorEvidence(content, &context, &builder, .{}); + return builder.finish(); + } + + fn collectVectorEvidence(self: *Document, content: []const u8, context: *const ExtractionContext, builder: *vector_evidence.Builder, initial: vector_evidence.Graphics) anyerror!void { + // Bound cumulative input to the additional vector lexer, including + // repeated Form placements. Earlier PDF decompression is independent. + if (content.len > builder.limits.max_content_bytes - builder.content_bytes) { + builder.limited(true); + return; + } + builder.content_bytes += @intCast(content.len); + var token_arena = std.heap.ArenaAllocator.init(context.scratch_allocator); + defer token_arena.deinit(); + var lexer = interpreter.ContentLexer.init(token_arena.allocator(), content); + var operands: [128]interpreter.Operand = undefined; + var count: usize = 0; + var collector = vector_evidence.Collector{ .builder = builder, .graphics = initial, .scope = context.scope }; + while (!builder.stopped) { + const token_start = lexer.pos; + const token = (lexer.next() catch |err| { + if (err == error.OutOfMemory) return err; + builder.unknown(); + break; + }) orelse break; + if (token == .number and !vectorNumberTokenIsValid(content[token_start..lexer.pos], token.number)) { + builder.unknown(); + break; + } + if (token != .operator) { + if (count == operands.len) { + builder.limited(true); + break; + } + _ = pushOperand(&operands, &count, token); + continue; + } + if (!builder.operator()) break; + const op = token.operator; + if (std.mem.eql(u8, op, "Do")) { + if (count != 1 or operands[0] != .name) { + builder.unknown(); + } else self.collectVectorForm(operands[0].name, context, builder, collector.graphics) catch |err| { + if (err == error.OutOfMemory) return err; + if (err == error.FormDepthLimitExceeded or err == error.FormInvocationLimitExceeded) builder.limited(true) else builder.unknown(); + }; + } else try collector.consume(op, operands[0..count]); + count = 0; + // No shape borrows operands. Bound temporary decoded strings to one + // operator rather than retaining every text token on the page. + _ = token_arena.reset(.retain_capacity); + } + if (count != 0 and !builder.stopped) builder.unknown(); + collector.endStream(); + } + + fn vectorNumberTokenIsValid(raw: []const u8, value: f64) bool { + // The shared permissive lexer can replace an invalid numeric token + // with zero. Validate its original bytes before using path geometry. + if (!std.math.isFinite(value) or raw.len == 0) return false; + var start = raw.len; + while (start > 0 and raw[start - 1] != 0 and !std.ascii.isWhitespace(raw[start - 1])) start -= 1; + const parsed = std.fmt.parseFloat(f64, raw[start..]) catch return false; + return std.math.isFinite(parsed); + } + + fn collectVectorForm(self: *Document, name: []const u8, context: *const ExtractionContext, builder: *vector_evidence.Builder, parent: vector_evidence.Graphics) anyerror!void { + // Image Do is deliberately excluded from painted-path evidence. + const parent_resources = context.resources orelse return error.MissingVectorResources; + const xobjects = try resolveContextDict(context, parent_resources.get("XObject") orelse return error.MissingVectorXObjects); + const object = xobjects.get(name) orelse return error.MissingFormXObject; + const form = (try resolveForm(context, name)) orelse { + const resolved = try resolveContextObject(context, object); + if (resolved == .stream and std.mem.eql(u8, resolved.stream.dict.getName("Subtype") orelse "", "Image")) return; + return error.UnsupportedVectorXObject; + }; + const traversal = context.traversal orelse return error.MissingFormTraversal; + const scope = try traversal.enter(.{ .object_num = form.reference.num, .generation = form.reference.gen }); + defer traversal.leave(); + var arena = std.heap.ArenaAllocator.init(context.scratch_allocator); + defer arena.deinit(); + const scratch = arena.allocator(); + const content = try decompress.decompressStream(scratch, form.stream.data, form.stream.dict.get("Filter"), form.stream.dict.get("DecodeParms")); + var child = context.*; + child.scratch_allocator = scratch; + child.scope = scope; + if (form.stream.dict.get("Resources")) |resources| child.resources = try resolveContextDict(context, resources); + var graphics = parent; + graphics.paint_state = try formGraphicsState(parent.paint_state, form.stream.dict); + if (graphics.paint_state.clip_unknown and !parent.paint_state.clip_unknown) builder.unknown(); + try self.collectVectorEvidence(content, &child, builder, graphics); + } + pub fn getPageImagesFromContent( self: *Document, page_idx: usize, @@ -5844,6 +5951,108 @@ fn formTraversalPdfWithPageFont(allocator: std.mem.Allocator, page_content: []co return pdf.toOwnedSlice(allocator); } +test "vector evidence Form placements apply matrix once and preserve scoped clip" { + const allocator = std.testing.allocator; + const bytes = try formTraversalPdf(allocator, "q 1 0 0 1 20 50 cm /Fm Do Q q 1 0 0 1 100 50 cm /Fm Do Q /Im Do", "1 2 3 4 re f 200 200 2 2 re f", "/Resources 7 0 R /Matrix [2 0 0 3 10 20]"); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + var evidence = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "q 1 0 0 1 20 50 cm /Fm Do Q q 1 0 0 1 100 50 cm /Fm Do Q /Im Do"); + defer evidence.deinit(); + try std.testing.expectEqual(.complete, evidence.status); + try std.testing.expectEqual(@as(usize, 4), evidence.shapes.len); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 32, .y0 = 76, .x1 = 38, .y1 = 88 }, evidence.shapes[0].bbox); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 112, .y0 = 76, .x1 = 118, .y1 = 88 }, evidence.shapes[2].bbox); + try std.testing.expectEqual(.inside, evidence.shapes[0].clip_state); + try std.testing.expectEqual(.outside, evidence.shapes[1].clip_state); + try std.testing.expectEqual(@as(?u32, 6), evidence.shapes[0].content_scope.?.stream_object_num); + try std.testing.expect(evidence.shapes[0].content_scope.?.invocation_index != evidence.shapes[2].content_scope.?.invocation_index); +} + +test "vector evidence Form cycles are partial and invocation operator caps are shared" { + const allocator = std.testing.allocator; + const bytes = try formTraversalPdf(allocator, "/Fm Do", "0 0 2 2 re f /Fm Do", "/Resources 7 0 R"); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + var cycle = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "/Fm Do"); + defer cycle.deinit(); + try std.testing.expectEqual(.partial, cycle.status); + try std.testing.expectEqual(@as(usize, 1), cycle.shapes.len); + var limited = try doc.getPageVectorEvidenceFromContentWithLimits(0, allocator, allocator, "/Fm Do /Fm Do", .{ .max_form_invocations = 1 }); + defer limited.deinit(); + try std.testing.expectEqual(.limited, limited.status); + try std.testing.expectEqual(@as(usize, 1), limited.shapes.len); + var ops = try doc.getPageVectorEvidenceFromContentWithLimits(0, allocator, allocator, "/Fm Do", .{ .max_operators = 2 }); + defer ops.deinit(); + try std.testing.expectEqual(.limited, ops.status); + try std.testing.expectEqual(@as(u32, 2), ops.operator_count); + try std.testing.expectEqual(@as(usize, 0), ops.shapes.len); + var bytes_limited = try doc.getPageVectorEvidenceFromContentWithLimits(0, allocator, allocator, "/Fm Do", .{ .max_content_bytes = 6 }); + defer bytes_limited.deinit(); + try std.testing.expectEqual(.limited, bytes_limited.status); + try std.testing.expectEqual(@as(u32, 6), bytes_limited.content_bytes); + try std.testing.expectEqual(@as(usize, 0), bytes_limited.shapes.len); +} + +fn vectorFormAllocationProbe(allocator: std.mem.Allocator, doc: *Document) !void { + var evidence = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "q 1 0 0 1 20 50 cm /Fm Do Q /Fm Do"); + defer evidence.deinit(); + try std.testing.expectEqual(@as(usize, 2), evidence.shapes.len); +} + +test "vector evidence unresolved Do is partial while known images are ignored" { + const allocator = std.testing.allocator; + const bytes = try formTraversalPdf(allocator, "/Missing Do /Im Do", "0 0 2 2 re f", ""); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + var missing = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "/Missing Do"); + defer missing.deinit(); + try std.testing.expectEqual(.partial, missing.status); + try std.testing.expectEqual(@as(u32, 1), missing.unknown_count); + var image = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "/Im Do"); + defer image.deinit(); + try std.testing.expectEqual(.complete, image.status); + try std.testing.expectEqual(@as(usize, 0), image.shapes.len); + const resources = doc.pages.items[0].resources; + defer doc.pages.items[0].resources = resources; + doc.pages.items[0].resources = null; + var absent = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "/Fm Do"); + defer absent.deinit(); + try std.testing.expectEqual(.partial, absent.status); + try std.testing.expectEqual(@as(u32, 1), absent.unknown_count); +} + +test "vector evidence malformed numeric bytes cannot become synthetic zero coordinates" { + const allocator = std.testing.allocator; + const bytes = try formTraversalPdf(allocator, "", "", ""); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + for ([_][]const u8{ "+ 0 2 3 re f", "0 0 - 3 re f", "0 0 . 3 re f" }) |content| { + var evidence = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, content); + defer evidence.deinit(); + try std.testing.expectEqual(.partial, evidence.status); + try std.testing.expectEqual(@as(usize, 0), evidence.shapes.len); + } + var valid = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "% ignored numeric comment 123\n -1 .5 2 3 re f"); + defer valid.deinit(); + try std.testing.expectEqual(.complete, valid.status); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = -1, .y0 = 0.5, .x1 = 1, .y1 = 3.5 }, valid.shapes[0].bbox); +} + +test "vector evidence Form frames free partial output at every allocation failure" { + const allocator = std.testing.allocator; + const bytes = try formTraversalPdf(allocator, "/Fm Do", "1 2 3 4 re f", "/Resources 7 0 R /Matrix [2 0 0 3 10 20]"); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + var warmup = try doc.getPageVectorEvidenceFromContent(0, allocator, allocator, "/Fm Do"); + warmup.deinit(); + try std.testing.checkAllAllocationFailures(allocator, vectorFormAllocationProbe, .{doc}); +} + test "Form traversal resolves scoped named ActualText and preserves inherited page resources" { const allocator = std.testing.allocator; for ([_][]const u8{ "", "/Resources 7 0 R" }) |resources| { @@ -6598,6 +6807,79 @@ test "search candidate union ownership is safe across every allocation failure" ); } +fn pageGeometryDocumentFixture(allocator: std.mem.Allocator) ![]u8 { + var bytes: std.ArrayList(u8) = .empty; + errdefer bytes.deinit(allocator); + var writer = runtime.arrayListWriter(&bytes, allocator); + const objects = [_][]const u8{ + "<< /Type /Catalog /Pages 2 0 R >>", + "<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 /MediaBox [10 20 110 220] /CropBox [25 40 100 200] /Rotate 90 /UserUnit 3 >>", + "<< /Type /Page /Parent 2 0 R /UserUnit 5 0 R /Rotate 0 >>", + "<< /Type /Page /Parent 2 0 R >>", + "2", + }; + var offsets: [objects.len]usize = undefined; + try writer.writeAll("%PDF-1.7\n"); + for (objects, 0..) |object, index| { + offsets[index] = bytes.items.len; + try writer.print("{d} 0 obj\n{s}\nendobj\n", .{ index + 1, object }); + } + const xref_offset = bytes.items.len; + try writer.print("xref\n0 {d}\n0000000000 65535 f \n", .{objects.len + 1}); + for (offsets) |offset| try writer.print("{d:0>10} 00000 n \n", .{offset}); + try writer.print("trailer\n<< /Size {d} /Root 1 0 R >>\nstartxref\n{d}\n%%EOF\n", .{ objects.len + 1, xref_offset }); + return bytes.toOwnedSlice(allocator); +} + +test "document page geometry preserves raw origins inherited boxes and leaf UserUnit" { + const allocator = std.testing.allocator; + const bytes = try pageGeometryDocumentFixture(allocator); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + try std.testing.expectEqual(@as(usize, 2), doc.pageCount()); + const first = doc.getPageGeometry(0).?; + const second = doc.getPageGeometry(1).?; + try std.testing.expectEqual(@as(f64, 2), first.user_unit); + try std.testing.expectEqual(@as(f64, 1), second.user_unit); + try std.testing.expectEqual(page_geometry.QuarterTurn.clockwise_90, second.rotation); + try std.testing.expectEqual(layout.BBox{ .x0 = 10, .y0 = 20, .x1 = 110, .y1 = 220 }, first.media_box); + try std.testing.expectEqual(layout.BBox{ .x0 = 25, .y0 = 40, .x1 = 100, .y1 = 200 }, first.crop_box); + try std.testing.expectEqual(@as(f64, 100), doc.getPageInfo(0).?.width); + try std.testing.expectEqual(@as(f64, 200), doc.getPageInfo(0).?.height); + try std.testing.expectEqual(@as(f64, 200), first.media.width_points); + try std.testing.expectEqual(@as(f64, 400), first.media.height_points); + try std.testing.expect(doc.getPageGeometry(2) == null); +} + +test "document page geometry literal quarter transforms and unsupported geometry" { + const allocator = std.testing.allocator; + const bytes = try pageGeometryDocumentFixture(allocator); + defer allocator.free(bytes); + const doc = try Document.openFromMemory(allocator, bytes, ErrorConfig.default()); + defer doc.close(); + const point: page_geometry.Point = .{ .x = 40, .y = 50 }; + const expected_media = [_]page_geometry.Point{ .{ .x = 60, .y = 340 }, .{ .x = 60, .y = 60 }, .{ .x = 140, .y = 60 }, .{ .x = 340, .y = 140 } }; + const expected_crop = [_]page_geometry.Point{ .{ .x = 30, .y = 300 }, .{ .x = 20, .y = 30 }, .{ .x = 120, .y = 20 }, .{ .x = 300, .y = 120 } }; + for ([_]i32{ 0, 90, 180, 270 }, 0..) |rotation, index| { + doc.pages.items[0].rotation = rotation; + const model = doc.getPageGeometry(0).?; + try std.testing.expectEqual(expected_media[index], try page_geometry.mapPoint(model.pdfToDisplay(.media), point)); + try std.testing.expectEqual(expected_crop[index], try page_geometry.mapPoint(model.pdfToDisplay(.crop), point)); + try std.testing.expectEqual(point, try page_geometry.mapPoint(model.displayToPdf(.crop), expected_crop[index])); + } + doc.pages.items[0].rotation = 45; + try std.testing.expect(doc.getPageGeometry(0) == null); + doc.pages.items[0].rotation = 0; + for ([_]f64{ 0, -1, std.math.nan(f64), std.math.inf(f64), 75001 }) |unit| { + doc.pages.items[0].user_unit = unit; + try std.testing.expect(doc.getPageGeometry(0) == null); + } + doc.pages.items[0].user_unit = 1; + doc.pages.items[0].crop_box = .{ 300, 400, 400, 500 }; + try std.testing.expect(doc.getPageGeometry(0) == null); +} + test "formula crop geometry is padded clamped and top-left mapped" { const geometry = Document.formulaCropGeometry( .{ .x0 = 0, .y0 = 0, .x1 = 612, .y1 = 792 }, @@ -6644,7 +6926,7 @@ test "regional OCR preflight preserves unsupported and bounded outcomes" { defer memory.deinit(allocator); try std.testing.expectEqual(ocr.DiagnosticCode.rasterizer_requires_file, memory.diagnostic_code.?); doc.source_path = try allocator.dupe(u8, "unused.pdf"); - doc.pages.items[0].rotation = 90; + doc.pages.items[0].rotation = 45; var rotated = try doc.rasterizeOcrRegionDetailed(allocator, 0, bbox, .{}); defer rotated.deinit(allocator); try std.testing.expectEqual(ocr.DiagnosticCode.rasterizer_rotation_unsupported, rotated.diagnostic_code.?); diff --git a/src/schema.zig b/src/schema.zig index 7929eb7..667cf9f 100644 --- a/src/schema.zig +++ b/src/schema.zig @@ -1494,6 +1494,10 @@ fn writePageRouteRecord(writer: anytype, document_id: []const u8, source_id: ?[] try writer.print(",\"route_trace_id\":\"route-page-{d}\"", .{route.page_index}); } try writer.print(",\"trace_kind\":\"page_route\",\"page_index\":{},\"region_index\":null,\"stage\":\"route_decision\"", .{route.page_index}); + try writer.writeAll(",\"page_geometry\":"); + try writePageGeometry(writer, if (@hasField(@TypeOf(route), "page_geometry")) route.page_geometry else null); + try writer.writeAll(",\"vector_evidence\":"); + try writePageVectorEvidence(writer, if (@hasField(@TypeOf(route), "vector_evidence")) route.vector_evidence else null); try writeRouteFields(writer, route.route, route.reason_mask, route.span_count, 0, route.image_count, route.char_count, route.signals, route.bbox); try writeCoverage(writer, assessment); try specialist_protocol.writeRouteTraceSpecialistFieldsWithReason(writer, route.route, route.reason_mask, route.page_index, null); @@ -1519,6 +1523,43 @@ fn writePageRouteRecord(writer: anytype, document_id: []const u8, source_id: ?[] try writer.writeByte('}'); } +fn writeExactBBox(writer: anytype, bbox: @import("layout.zig").BBox) !void { + try writer.print("{{\"x0\":{d},\"y0\":{d},\"x1\":{d},\"y1\":{d}}}", .{ bbox.x0, bbox.y0, bbox.x1, bbox.y1 }); +} + +fn writePageVectorEvidence(writer: anytype, value: ?@import("vector_routing.zig").Summary) !void { + const evidence = value orelse return writer.writeAll("null"); + try writer.print("{{\"status\":\"{s}\",\"shape_count\":{},\"unknown_count\":{},\"ignored_rule_count\":{},\"clipped_out_count\":{},\"native_accounted_count\":{},\"simple_graphic_count\":{},\"ambiguous_shape_count\":{},\"candidate_region_count\":{},\"max_shapes\":{},\"max_segments_per_path\":{},\"max_operators\":{},\"max_candidate_regions\":{},\"operator_count\":{},\"content_bytes\":{},\"max_content_bytes\":{},\"max_graphics_depth\":{},\"max_form_invocations\":{}}}", .{ + @tagName(evidence.status), evidence.shape_count, evidence.unknown_count, evidence.ignored_rule_count, + evidence.clipped_out_count, evidence.native_accounted_count, evidence.simple_graphic_count, evidence.ambiguous_shape_count, + evidence.candidate_region_count, evidence.max_shapes, evidence.max_segments_per_path, evidence.max_operators, + evidence.max_candidate_regions, evidence.operator_count, evidence.content_bytes, evidence.max_content_bytes, + evidence.max_graphics_depth, evidence.max_form_invocations, + }); +} + +fn writeRegionVectorEvidence(writer: anytype, value: ?@import("vector_routing.zig").Candidate) !void { + const evidence = value orelse return writer.writeAll("null"); + try writer.print("{{\"shape_count\":{},\"segment_count\":{},\"subpath_count\":{},\"native_coverage\":{d:.6},\"decision\":\"unexplained_vector\",\"bbox\":", .{ + evidence.shape_count, evidence.segment_count, evidence.subpath_count, evidence.native_coverage, + }); + try writeExactBBox(writer, evidence.bbox); + try writer.writeByte('}'); +} + +fn writePageGeometry(writer: anytype, value: ?@import("page_geometry.zig").PageGeometry) !void { + const geometry = value orelse return writer.writeAll("null"); + try writer.print("{{\"coordinate_space\":\"pdf_user_space\",\"user_unit\":{d},\"rotation_degrees\":{},\"media_box\":", .{ geometry.user_unit, @backingInt(geometry.rotation) }); + try writeExactBBox(writer, geometry.media_box); + try writer.writeAll(",\"crop_box\":"); + try writeExactBBox(writer, geometry.crop_box); + try writer.print(",\"display\":{{\"box\":\"crop_box\",\"origin\":\"top_left\",\"unit\":\"point\",\"width\":{d},\"height\":{d},\"pdf_to_display\":", .{ geometry.crop.width_points, geometry.crop.height_points }); + try specialist_protocol.writeMatrix(writer, geometry.crop.pdf_to_display); + try writer.writeAll(",\"display_to_pdf\":"); + try specialist_protocol.writeMatrix(writer, geometry.crop.display_to_pdf); + try writer.writeAll("}}"); +} + fn writeRegionRouteRecord(writer: anytype, document_id: []const u8, source_id: ?[]const u8, input_sha256: ?[]const u8, route: anytype, route_index: u32, stream_meta: ?StreamRecordMeta, assessment: coverage.Assessment) !void { try writeRecordHeader(writer, "route_trace"); if (stream_meta) |meta| try writeStreamMeta(writer, meta); @@ -1530,6 +1571,8 @@ fn writeRegionRouteRecord(writer: anytype, document_id: []const u8, source_id: ? try writer.print(",\"route_trace_id\":\"route-region-{d}\"", .{route.region_index}); } try writer.print(",\"trace_kind\":\"region_route\",\"page_index\":{},\"region_index\":{},\"stage\":\"route_decision\"", .{ route.page_index, route.region_index }); + try writer.writeAll(",\"vector_evidence\":"); + try writeRegionVectorEvidence(writer, if (@hasField(@TypeOf(route), "vector_evidence")) route.vector_evidence else null); try writer.writeAll(",\"layout_block_index\":"); try writeOptionalU32(writer, route.layout_block_index); try writer.writeAll(",\"block_kind\":"); @@ -2427,6 +2470,8 @@ fn writeReasonArray(writer: anytype, mask: u32) !void { .{ .bit = 10, .name = "layout_route_stub" }, .{ .bit = 11, .name = "table_route_stub" }, .{ .bit = 12, .name = "formula_route_stub" }, + .{ .bit = 13, .name = "unexplained_vector" }, + .{ .bit = 14, .name = "vector_evidence_limited" }, }; try writer.writeByte('['); diff --git a/src/specialist_protocol.zig b/src/specialist_protocol.zig index 9c98aea..8514585 100644 --- a/src/specialist_protocol.zig +++ b/src/specialist_protocol.zig @@ -10,8 +10,9 @@ const complexity = @import("complexity.zig"); const layout = @import("layout.zig"); const reconcile = @import("reconcile.zig"); const runtime = @import("runtime.zig"); +const page_geometry = @import("page_geometry.zig"); -pub const schema_version = "0.17.0"; +pub const schema_version = "0.19.0"; pub const SpecialistKind = enum { ocr, @@ -51,8 +52,35 @@ pub const RenderContext = struct { input_sha256: ?[]const u8 = null, span_base: u32 = 0, block_base: u32 = 0, + crop_geometry: ?RasterContext = null, }; +pub const RasterContext = struct { + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, + pixel_width: u32, + pixel_height: u32, +}; + +pub fn writeMatrix(writer: anytype, matrix: page_geometry.Matrix) !void { + try writer.writeByte('['); + for (matrix, 0..) |value, index| { + if (index > 0) try writer.writeByte(','); + try writer.print("{d}", .{value}); + } + try writer.writeByte(']'); +} + +pub fn writeRasterContext(writer: anytype, context: ?RasterContext) !void { + const value = context orelse return writer.writeAll("null"); + const matrix = value.pixel_to_pdf orelse return writer.writeAll("null"); + try writer.writeAll("{\"pixel_to_pdf\":"); + try writeMatrix(writer, matrix); + try writer.writeAll(",\"rasterizer_dpi\":"); + if (value.rasterizer_dpi) |dpi| try writer.print("{d}", .{dpi}) else try writer.writeAll("null"); + try writer.print(",\"pixel_width\":{},\"pixel_height\":{}}}", .{ value.pixel_width, value.pixel_height }); +} + pub const StreamMeta = struct { event_index: u64, page_index: u32, @@ -436,7 +464,7 @@ pub fn writeEntityRequestForBlock( try writer.print("\"block-{d}\"", .{context.block_base + block.id}); try writer.writeAll("],\"blocks\":["); try writeBlockContext(writer, block, context.block_base + block.id); - try writer.writeAll("],\"ruling_lines\":[],\"crop_image_path\":null,\"debug_asset_ids\":[]"); + try writer.writeAll("],\"ruling_lines\":[],\"crop_image_path\":null,\"crop_geometry\":null,\"debug_asset_ids\":[]"); try writeProvenance(writer, context, "specialist-request-entity", block.page_index, block.bbox, "lifecycle", 1.0); try writer.writeByte('}'); } @@ -606,6 +634,13 @@ fn writeFormulaAttempts( if (attempt.crop_pixel_width) |width| try writer.print("{}", .{width}) else try writer.writeAll("null"); try writer.writeAll(",\"crop_pixel_height\":"); if (attempt.crop_pixel_height) |height| try writer.print("{}", .{height}) else try writer.writeAll("null"); + try writer.writeAll(",\"crop_transform\":"); + try writeRasterContext(writer, .{ + .pixel_to_pdf = if (@hasField(@TypeOf(attempt), "crop_pixel_to_pdf")) attempt.crop_pixel_to_pdf else null, + .rasterizer_dpi = if (@hasField(@TypeOf(attempt), "crop_rasterizer_dpi")) attempt.crop_rasterizer_dpi else null, + .pixel_width = attempt.crop_pixel_width orelse 0, + .pixel_height = attempt.crop_pixel_height orelse 0, + }); try writer.print("}},\"execution\":{{\"duration_ms\":{},\"exit_code\":", .{attempt.duration_ms}); if (attempt.exit_code) |code| try writer.print("{}", .{code}) else try writer.writeAll("null"); try writer.writeAll("},\"quality\":{\"formula_count\":"); @@ -838,6 +873,8 @@ fn writeRequestRecord( } else { try writer.writeAll("null"); } + try writer.writeAll(",\"crop_geometry\":"); + try writeRasterContext(writer, context.crop_geometry); try writer.writeAll(",\"debug_asset_ids\":"); try writeDebugAssetIds(writer, page_index); try writeProvenance(writer, context, "specialist-request", page_index, bbox, "lifecycle", routeConfidence(route)); @@ -886,6 +923,13 @@ fn writeOcrAttemptRecord(writer: anytype, context: RenderContext, attempt: anyty try writer.print(",\"max_image_pixels\":{},\"max_image_bytes\":{},\"max_regions_per_page\":{},\"max_attempts_per_page\":{}", .{ attempt.max_image_pixels, attempt.max_image_bytes, attempt.max_regions_per_page, attempt.max_attempts_per_page }); try writer.writeAll("},\"crop\":"); try writeOcrCrop(writer, attempt); + try writer.writeAll(",\"raster_transform\":"); + try writeRasterContext(writer, .{ + .pixel_to_pdf = if (@hasField(@TypeOf(attempt), "pixel_to_pdf")) attempt.pixel_to_pdf else null, + .rasterizer_dpi = if (@hasField(@TypeOf(attempt), "rasterizer_dpi")) attempt.rasterizer_dpi else null, + .pixel_width = attempt.pixel_width, + .pixel_height = attempt.pixel_height, + }); try writer.writeAll(",\"execution\":{\"duration_ms\":"); try writer.print("{}", .{attempt.duration_ms}); try writer.writeAll(",\"exit_code\":"); @@ -1490,6 +1534,8 @@ fn writeReasonArray(writer: anytype, mask: u32) !void { .{ .bit = 10, .name = "layout_route_stub" }, .{ .bit = 11, .name = "table_route_stub" }, .{ .bit = 12, .name = "formula_route_stub" }, + .{ .bit = 13, .name = "unexplained_vector" }, + .{ .bit = 14, .name = "vector_evidence_limited" }, }; try writer.writeByte('['); var first = true; diff --git a/src/specialists.zig b/src/specialists.zig index 05c4588..a6a20a9 100644 --- a/src/specialists.zig +++ b/src/specialists.zig @@ -7,6 +7,7 @@ const std = @import("std"); const layout = @import("layout.zig"); const runtime = @import("runtime.zig"); +const page_geometry = @import("page_geometry.zig"); pub const BBox = layout.BBox; pub const TextSpan = layout.TextSpan; @@ -125,6 +126,8 @@ pub const CropInput = struct { image_path: []const u8, pixel_width: u32 = 0, pixel_height: u32 = 0, + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, }; pub const TableSpecialistKind = enum { @@ -224,6 +227,8 @@ pub const FormulaCrop = struct { pixel_width: u32, pixel_height: u32, dpi: u32, + pixel_to_pdf: ?page_geometry.Matrix = null, + rasterizer_dpi: ?f64 = null, pub fn deinit(self: *FormulaCrop, allocator: std.mem.Allocator) void { runtime.deleteFilePath(self.image_path); diff --git a/src/vector_evidence.zig b/src/vector_evidence.zig new file mode 100644 index 0000000..4844db7 --- /dev/null +++ b/src/vector_evidence.zig @@ -0,0 +1,482 @@ +//! Bounded painted-path evidence. Boxes are conservative PDF-user-space hulls, +//! not rendered visibility or retained paths. +const std = @import("std"); +const layout = @import("layout.zig"); +const paint = @import("paint.zig"); +const geometry = @import("page_geometry.zig"); +const interpreter = @import("interpreter.zig"); +const form_scope = @import("form_scope.zig"); + +pub const Limits = struct { + max_shapes: u32 = 512, + max_segments_per_path: u32 = 512, + max_operators: u32 = 100_000, + max_graphics_depth: u32 = 32, + max_form_invocations: u32 = 256, + max_content_bytes: u32 = 8 * 1024 * 1024, +}; +pub const Status = enum { complete, partial, limited }; +pub const PaintKind = enum { fill, stroke, fill_stroke }; +pub const ClipState = enum { unclipped, inside, partial, outside, unknown }; +pub const Shape = struct { + bbox: layout.BBox, + paint_kind: PaintKind, + rectangle_only: bool, + segment_count: u32, + subpath_count: u32, + clip_state: ClipState, + content_scope: ?form_scope.Scope, +}; +pub const Evidence = struct { + allocator: std.mem.Allocator, + shapes: []Shape, + status: Status, + operator_count: u32, + content_bytes: u32 = 0, + unknown_count: u32, + limits: Limits, + + pub fn deinit(self: *Evidence) void { + self.allocator.free(self.shapes); + self.* = undefined; + } +}; + +pub const Builder = struct { + allocator: std.mem.Allocator, + shapes: std.ArrayList(Shape) = .empty, + status: Status = .complete, + operator_count: u32 = 0, + content_bytes: u32 = 0, + unknown_count: u32 = 0, + limits: Limits = .{}, + stopped: bool = false, + + pub fn deinit(self: *Builder) void { + self.shapes.deinit(self.allocator); + } + pub fn finish(self: *Builder) !Evidence { + return .{ .allocator = self.allocator, .shapes = try self.shapes.toOwnedSlice(self.allocator), .status = self.status, .operator_count = self.operator_count, .content_bytes = self.content_bytes, .unknown_count = self.unknown_count, .limits = self.limits }; + } + pub fn unknown(self: *Builder) void { + if (self.status == .complete) self.status = .partial; + self.unknown_count +|= 1; + } + pub fn limited(self: *Builder, stop: bool) void { + self.status = .limited; + self.stopped = self.stopped or stop; + } + pub fn operator(self: *Builder) bool { + if (self.stopped) return false; + if (self.operator_count >= self.limits.max_operators) { + self.limited(true); + return false; + } + self.operator_count += 1; + return true; + } +}; + +pub const Graphics = struct { + paint_state: paint.State = .{}, + line_width: f64 = 1, + // Miter joins can extend beyond the control hull. + miter_limit: f64 = 10, + line_join: u8 = 0, +}; + +pub const Collector = struct { + builder: *Builder, + graphics: Graphics = .{}, + scope: ?form_scope.Scope = null, + stack: [32]Graphics = undefined, + depth: usize = 0, + bbox: ?layout.BBox = null, + segments: u32 = 0, + subpaths: u32 = 0, + rectangle_only: bool = true, + path_invalid: bool = false, + current_point: bool = false, + pending_clip: bool = false, + clip_exact_rectangle: bool = true, + text_clip_pending: bool = false, + + fn invalidate(self: *Collector) void { + self.builder.unknown(); + self.path_invalid = true; + } + fn numbers(operands: []const interpreter.Operand, count: usize) ?[6]f64 { + if (operands.len != count) return null; + var values: [6]f64 = @splat(0); + for (operands, 0..) |operand, index| { + if (operand != .number or !std.math.isFinite(operand.number)) return null; + values[index] = operand.number; + } + return values; + } + fn point(self: *Collector, x: f64, y: f64) void { + if (!self.graphics.paint_state.geometry_known) { + self.path_invalid = true; + return; + } + const p = geometry.mapPoint(self.graphics.paint_state.ctm, .{ .x = x, .y = y }) catch { + self.invalidate(); + return; + }; + if (self.bbox) |*box| { + box.x0 = @min(box.x0, p.x); + box.y0 = @min(box.y0, p.y); + box.x1 = @max(box.x1, p.x); + box.y1 = @max(box.y1, p.y); + } else self.bbox = .{ .x0 = p.x, .y0 = p.y, .x1 = p.x, .y1 = p.y }; + } + fn segment(self: *Collector, count: u32) bool { + if (count > self.builder.limits.max_segments_per_path -| self.segments) { + self.builder.limited(false); + self.path_invalid = true; + return false; + } + self.segments += count; + return true; + } + fn resetPath(self: *Collector) void { + self.bbox = null; + self.segments = 0; + self.subpaths = 0; + self.rectangle_only = true; + self.path_invalid = false; + self.current_point = false; + self.pending_clip = false; + self.clip_exact_rectangle = true; + } + fn endPath(self: *Collector, kind: ?PaintKind) !void { + // Painting uses the previous clip; W is applied only at path end. + if (kind) |paint_kind| { + if (!self.path_invalid and self.segments > 0 and self.bbox != null) { + var box = self.bbox.?; + if (paint_kind != .fill) { + const m = self.graphics.paint_state.ctm; + const join_scale = if (self.graphics.line_join == 0 and self.segments > 2) @max(1, self.graphics.miter_limit) else 1; + const radius = self.graphics.line_width * 0.5 * join_scale; + const dx = radius * (@abs(m[0]) + @abs(m[2])); + const dy = radius * (@abs(m[1]) + @abs(m[3])); + box = .{ .x0 = box.x0 - dx, .y0 = box.y0 - dy, .x1 = box.x1 + dx, .y1 = box.y1 + dy }; + } + if (!std.math.isFinite(box.x0) or !std.math.isFinite(box.y0) or !std.math.isFinite(box.x1) or !std.math.isFinite(box.y1)) { + self.builder.unknown(); + } else if (self.builder.shapes.items.len >= self.builder.limits.max_shapes) { + self.builder.limited(true); + } else { + const clip = self.graphics.paint_state.evidence(box).clip_state; + if (clip == .partial) { + if (self.graphics.paint_state.clip_bbox) |bounds| { + box = .{ .x0 = @max(box.x0, bounds.x0), .y0 = @max(box.y0, bounds.y0), .x1 = @min(box.x1, bounds.x1), .y1 = @min(box.y1, bounds.y1) }; + } + } + try self.builder.shapes.append(self.builder.allocator, .{ .bbox = box, .paint_kind = paint_kind, .rectangle_only = self.rectangle_only, .segment_count = self.segments, .subpath_count = self.subpaths, .clip_state = switch (clip) { + .unclipped => .unclipped, + .inside => .inside, + .outside => .outside, + .partial => .partial, + .unknown => .unknown, + }, .content_scope = self.scope }); + } + } + } + if (self.pending_clip) { + if (self.path_invalid or self.bbox == null or !self.clip_exact_rectangle or self.subpaths != 1 or !self.rectangle_only) { + self.graphics.paint_state.clip_unknown = true; + self.graphics.paint_state.clip_bbox = null; + self.builder.unknown(); + } else if (!self.graphics.paint_state.clip_unknown) { + var box = self.bbox.?; + if (self.graphics.paint_state.clip_bbox) |old| { + box.x0 = @max(box.x0, old.x0); + box.y0 = @max(box.y0, old.y0); + box.x1 = @max(box.x0, @min(box.x1, old.x1)); + box.y1 = @max(box.y0, @min(box.y1, old.y1)); + } + self.graphics.paint_state.clip_bbox = box; + } + } + self.resetPath(); + } + + /// Caller counts operators once, including Do handled outside this module. + pub fn consume(self: *Collector, op: []const u8, operands: []const interpreter.Operand) !void { + if (std.mem.eql(u8, op, "q")) { + if (operands.len != 0) { + self.invalidate(); + return; + } + if (self.depth >= @min(self.stack.len, self.builder.limits.max_graphics_depth)) { + self.builder.limited(true); + return; + } + self.stack[self.depth] = self.graphics; + self.depth += 1; + } else if (std.mem.eql(u8, op, "Q")) { + if (operands.len != 0 or self.depth == 0) { + self.invalidate(); + self.graphics.paint_state.geometry_known = false; + return; + } + self.depth -= 1; + self.graphics = self.stack[self.depth]; + } else if (std.mem.eql(u8, op, "cm")) { + const values = numbers(operands, 6) orelse { + self.invalidate(); + self.graphics.paint_state.geometry_known = false; + return; + }; + const matrix = paint.concat(self.graphics.paint_state.ctm, values); + for (matrix) |value| if (!std.math.isFinite(value)) { + self.invalidate(); + self.graphics.paint_state.geometry_known = false; + return; + }; + self.graphics.paint_state.ctm = matrix; + } else if (std.mem.eql(u8, op, "m") or std.mem.eql(u8, op, "l")) { + const values = numbers(operands, 2) orelse { + self.invalidate(); + return; + }; + const move = std.mem.eql(u8, op, "m"); + if (!move and !self.current_point) { + self.invalidate(); + return; + } + if (!self.segment(1)) return; + if (move) self.subpaths += 1; + self.current_point = true; + self.rectangle_only = false; + self.point(values[0], values[1]); + } else if (std.mem.eql(u8, op, "c") or std.mem.eql(u8, op, "v") or std.mem.eql(u8, op, "y")) { + const count: usize = if (std.mem.eql(u8, op, "c")) 6 else 4; + const values = numbers(operands, count) orelse { + self.invalidate(); + return; + }; + if (!self.current_point) { + self.invalidate(); + return; + } + if (!self.segment(1)) return; + self.rectangle_only = false; + var index: usize = 0; + while (index < count) : (index += 2) self.point(values[index], values[index + 1]); + } else if (std.mem.eql(u8, op, "re")) { + const values = numbers(operands, 4) orelse { + self.invalidate(); + return; + }; + if (!self.segment(4)) return; + self.subpaths += 1; + self.current_point = true; + const m = self.graphics.paint_state.ctm; + self.clip_exact_rectangle = self.clip_exact_rectangle and ((m[1] == 0 and m[2] == 0) or (m[0] == 0 and m[3] == 0)); + self.point(values[0], values[1]); + self.point(values[0] + values[2], values[1]); + self.point(values[0], values[1] + values[3]); + self.point(values[0] + values[2], values[1] + values[3]); + } else if (std.mem.eql(u8, op, "h")) { + if (operands.len != 0 or !self.current_point) { + self.invalidate(); + return; + } + _ = self.segment(1); + } else if (std.mem.eql(u8, op, "W") or std.mem.eql(u8, op, "W*")) { + if (operands.len != 0) self.invalidate(); + self.pending_clip = true; + } else if (std.mem.eql(u8, op, "n") or std.mem.eql(u8, op, "f") or std.mem.eql(u8, op, "F") or std.mem.eql(u8, op, "f*") or std.mem.eql(u8, op, "S") or std.mem.eql(u8, op, "s") or std.mem.eql(u8, op, "B") or std.mem.eql(u8, op, "B*") or std.mem.eql(u8, op, "b") or std.mem.eql(u8, op, "b*")) { + if (operands.len != 0) self.invalidate(); + if (std.mem.eql(u8, op, "s") or std.mem.eql(u8, op, "b") or std.mem.eql(u8, op, "b*")) { + if (!self.current_point) self.invalidate() else _ = self.segment(1); + } + const kind: ?PaintKind = if (std.mem.eql(u8, op, "n")) null else if (op[0] == 'S' or op[0] == 's') .stroke else if (op[0] == 'B' or op[0] == 'b') .fill_stroke else .fill; + try self.endPath(kind); + } else if (std.mem.eql(u8, op, "w") or std.mem.eql(u8, op, "M") or std.mem.eql(u8, op, "j")) { + const values = numbers(operands, 1) orelse { + self.invalidate(); + self.graphics.paint_state.geometry_known = false; + return; + }; + if (std.mem.eql(u8, op, "w")) { + if (values[0] <= 0) { + self.builder.unknown(); + self.graphics.paint_state.geometry_known = false; + } else self.graphics.line_width = values[0]; + } else if (std.mem.eql(u8, op, "M")) { + if (values[0] < 1) { + self.invalidate(); + self.graphics.paint_state.geometry_known = false; + } else self.graphics.miter_limit = values[0]; + } else { + if (values[0] < 0 or values[0] > 2 or @floor(values[0]) != values[0]) { + self.invalidate(); + self.graphics.paint_state.geometry_known = false; + } else self.graphics.line_join = @intFromFloat(values[0]); + } + } else if (std.mem.eql(u8, op, "gs")) { + self.builder.unknown(); + self.graphics.paint_state.geometry_known = false; + self.graphics.paint_state.clip_unknown = true; + } else if (std.mem.eql(u8, op, "Tr")) { + const values = numbers(operands, 1) orelse { + self.builder.unknown(); + self.graphics.paint_state.rendering_mode = null; + return; + }; + if (values[0] < 0 or values[0] > 7 or @floor(values[0]) != values[0]) { + self.builder.unknown(); + self.graphics.paint_state.rendering_mode = null; + } else self.graphics.paint_state.rendering_mode = @intFromFloat(values[0]); + } else if (std.mem.eql(u8, op, "Tj") or std.mem.eql(u8, op, "TJ") or std.mem.eql(u8, op, "'") or std.mem.eql(u8, op, "\"")) { + if (self.graphics.paint_state.rendering_mode == null or self.graphics.paint_state.rendering_mode.? >= 4) self.text_clip_pending = true; + } else if (std.mem.eql(u8, op, "ET")) { + if (self.text_clip_pending) { + self.graphics.paint_state.clip_unknown = true; + self.graphics.paint_state.clip_bbox = null; + self.builder.unknown(); + } + self.text_clip_pending = false; + } else { + const ignored = [_][]const u8{ "Do", "BT", "Tf", "Tm", "Td", "TD", "T*", "Tc", "Tw", "Tz", "TL", "Ts", "J", "d", "ri", "i", "CS", "cs", "SC", "SCN", "sc", "scn", "G", "g", "RG", "rg", "K", "k", "BMC", "BDC", "EMC", "MP", "DP", "BX", "EX" }; + for (ignored) |known| if (std.mem.eql(u8, op, known)) return; + self.builder.unknown(); + self.graphics.paint_state.geometry_known = false; + } + } + pub fn endStream(self: *Collector) void { + if (self.depth != 0 or self.bbox != null or self.pending_clip or self.text_clip_pending) self.builder.unknown(); + } +}; + +fn collectForTest(allocator: std.mem.Allocator, content: []const u8, limits: Limits) !Evidence { + var builder = Builder{ .allocator = allocator, .limits = limits }; + defer builder.deinit(); + var scratch = std.heap.ArenaAllocator.init(allocator); + defer scratch.deinit(); + var lexer = interpreter.ContentLexer.init(scratch.allocator(), content); + var operands: [128]interpreter.Operand = undefined; + var count: usize = 0; + var collector = Collector{ .builder = &builder }; + while (!builder.stopped) { + const token = (try lexer.next()) orelse break; + switch (token) { + .operator => |op| { + if (!builder.operator()) break; + try collector.consume(op, operands[0..count]); + count = 0; + }, + else => { + if (count == operands.len) { + builder.limited(true); + break; + } + operands[count] = switch (token) { + .number => |v| .{ .number = v }, + .name => |v| .{ .name = v }, + .string => |v| .{ .string = v }, + .hex_string => |v| .{ .hex_string = v }, + .array => |v| .{ .array = v }, + .operator => unreachable, + }; + count += 1; + }, + } + } + collector.endStream(); + return builder.finish(); +} + +test "vector evidence paints paths with transformed conservative cubic hulls" { + var evidence = try collectForTest(std.testing.allocator, "q 2 0 0 3 10 20 cm 0 0 m 5 10 15 -5 20 0 c h f Q 1 2 3 4 re B", .{}); + defer evidence.deinit(); + try std.testing.expectEqual(.complete, evidence.status); + try std.testing.expectEqual(@as(usize, 2), evidence.shapes.len); + const curve = evidence.shapes[0]; + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 10, .y0 = 5, .x1 = 50, .y1 = 50 }, curve.bbox); + try std.testing.expect(!curve.rectangle_only); + try std.testing.expectEqual(@as(u32, 3), curve.segment_count); + try std.testing.expectEqual(@as(u32, 1), curve.subpath_count); + try std.testing.expectEqual(.fill_stroke, evidence.shapes[1].paint_kind); + try std.testing.expect(evidence.shapes[1].rectangle_only); +} + +test "single stroked segments have no miter join expansion" { + var evidence = try collectForTest(std.testing.allocator, "0.5 w 40 155 m 360 155 l S", .{}); + defer evidence.deinit(); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 39.75, .y0 = 154.75, .x1 = 360.25, .y1 = 155.25 }, evidence.shapes[0].bbox); +} + +test "vector evidence clipping-only paths emit no shape and graphics restore clip" { + var evidence = try collectForTest(std.testing.allocator, "q 0 0 20 20 re W n 2 2 3 3 re f 30 30 2 2 re f 18 18 5 5 re f Q 30 30 2 2 re f", .{}); + defer evidence.deinit(); + try std.testing.expectEqual(.complete, evidence.status); + try std.testing.expectEqual(@as(usize, 4), evidence.shapes.len); + try std.testing.expectEqual(.inside, evidence.shapes[0].clip_state); + try std.testing.expectEqual(.outside, evidence.shapes[1].clip_state); + try std.testing.expectEqual(.partial, evidence.shapes[2].clip_state); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 18, .y0 = 18, .x1 = 20, .y1 = 20 }, evidence.shapes[2].bbox); + try std.testing.expectEqual(.unclipped, evidence.shapes[3].clip_state); + var uncertain = try collectForTest(std.testing.allocator, "0 0 m 20 0 l 10 20 l h W* n 2 2 3 3 re f", .{}); + defer uncertain.deinit(); + try std.testing.expectEqual(.partial, uncertain.status); + try std.testing.expectEqual(.unknown, uncertain.shapes[0].clip_state); +} + +test "vector evidence malformed and unknown graphics withhold invented geometry" { + var evidence = try collectForTest(std.testing.allocator, "0 0 m /bad 1 l f q /Alpha gs 0 0 2 2 re f Q 10 20 2 3 re f", .{}); + defer evidence.deinit(); + try std.testing.expectEqual(.partial, evidence.status); + try std.testing.expectEqual(@as(usize, 1), evidence.shapes.len); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 10, .y0 = 20, .x1 = 12, .y1 = 23 }, evidence.shapes[0].bbox); + var builder = Builder{ .allocator = std.testing.allocator }; + defer builder.deinit(); + var collector = Collector{ .builder = &builder }; + try collector.consume("re", &.{ .{ .number = std.math.inf(f64) }, .{ .number = 0 }, .{ .number = 2 }, .{ .number = 3 } }); + try collector.consume("f", &.{}); + try std.testing.expectEqual(.partial, builder.status); + try std.testing.expectEqual(@as(usize, 0), builder.shapes.items.len); +} + +test "vector evidence resource caps preserve only bounded painted prefixes" { + var shapes = try collectForTest(std.testing.allocator, "0 0 2 2 re f 3 3 2 2 re f", .{ .max_shapes = 1 }); + defer shapes.deinit(); + try std.testing.expectEqual(.limited, shapes.status); + try std.testing.expectEqual(@as(usize, 1), shapes.shapes.len); + var segments = try collectForTest(std.testing.allocator, "0 0 m 1 1 l 2 2 l f 3 3 m 4 4 l S", .{ .max_segments_per_path = 2 }); + defer segments.deinit(); + try std.testing.expectEqual(.limited, segments.status); + try std.testing.expectEqual(@as(usize, 1), segments.shapes.len); + var ops = try collectForTest(std.testing.allocator, "0 0 2 2 re f 3 3 2 2 re f", .{ .max_operators = 2 }); + defer ops.deinit(); + try std.testing.expectEqual(.limited, ops.status); + try std.testing.expectEqual(@as(u32, 2), ops.operator_count); + try std.testing.expectEqual(@as(usize, 1), ops.shapes.len); + var stack = try collectForTest(std.testing.allocator, "q q 0 0 2 2 re f Q Q", .{ .max_graphics_depth = 1 }); + defer stack.deinit(); + try std.testing.expectEqual(.limited, stack.status); + try std.testing.expectEqual(@as(usize, 0), stack.shapes.len); +} + +test "vector evidence invalid stroke style persists past path end until graphics restore" { + for ([_][]const u8{ "q /bad w n 0 0 2 2 re S Q 10 20 2 3 re f", "q -1 w n 0 0 2 2 re S Q 10 20 2 3 re f", "q 0 M n 0 0 2 2 re S Q 10 20 2 3 re f", "q 3 j n 0 0 2 2 re S Q 10 20 2 3 re f" }) |content| { + var evidence = try collectForTest(std.testing.allocator, content, .{}); + defer evidence.deinit(); + try std.testing.expectEqual(.partial, evidence.status); + try std.testing.expectEqual(@as(usize, 1), evidence.shapes.len); + try std.testing.expectEqualDeep(layout.BBox{ .x0 = 10, .y0 = 20, .x1 = 12, .y1 = 23 }, evidence.shapes[0].bbox); + } +} + +fn vectorAllocationProbe(allocator: std.mem.Allocator) !void { + var evidence = try collectForTest(allocator, "q 2 0 0 3 10 20 cm 0 0 20 20 re f Q 0 0 m 2 4 3 4 5 0 c S", .{}); + defer evidence.deinit(); + try std.testing.expectEqual(@as(usize, 2), evidence.shapes.len); +} + +test "vector evidence frees partial allocations at every allocation failure" { + try std.testing.checkAllAllocationFailures(std.testing.allocator, vectorAllocationProbe, .{}); +} diff --git a/src/vector_routing.zig b/src/vector_routing.zig new file mode 100644 index 0000000..f71814f --- /dev/null +++ b/src/vector_routing.zig @@ -0,0 +1,317 @@ +//! Bounded, conservative candidates for unexplained painted vector lettering. +//! Geometry suggests OCR targets; it does not establish text or visibility. +const std = @import("std"); +const layout = @import("layout.zig"); +const geometry = @import("page_geometry.zig"); +const vector = @import("vector_evidence.zig"); + +pub const Summary = struct { + status: vector.Status = .complete, + shape_count: u32 = 0, + unknown_count: u32 = 0, + ignored_rule_count: u32 = 0, + clipped_out_count: u32 = 0, + native_accounted_count: u32 = 0, + simple_graphic_count: u32 = 0, + ambiguous_shape_count: u32 = 0, + candidate_region_count: u32 = 0, + max_shapes: u32 = 512, + max_segments_per_path: u32 = 512, + max_operators: u32 = 100_000, + operator_count: u32 = 0, + content_bytes: u32 = 0, + max_content_bytes: u32 = 8 * 1024 * 1024, + max_graphics_depth: u32 = 32, + max_form_invocations: u32 = 256, + max_candidate_regions: u32 = 64, +}; +pub const Candidate = struct { + bbox: layout.BBox, + shape_count: u32, + segment_count: u32, + subpath_count: u32, + native_coverage: f64, + decision: enum { unexplained_vector } = .unexplained_vector, +}; +pub const Result = struct { + allocator: std.mem.Allocator, + summary: Summary, + candidates: []Candidate, + pub fn deinit(self: *Result) void { + self.allocator.free(self.candidates); + self.* = undefined; + } +}; + +pub fn mergeCandidates(a: Candidate, b: Candidate, spans: []const layout.TextSpan) ?Candidate { + const box = unionBox(a.bbox, b.bbox); + var checks: usize = 0; + const coverage = nativeCoverage(box, spans, &checks); + if (!coverage.complete) return null; + return .{ .bbox = box, .shape_count = a.shape_count + b.shape_count, .segment_count = a.segment_count + b.segment_count, .subpath_count = a.subpath_count + b.subpath_count, .native_coverage = coverage.fraction }; +} +const max_shapes = 512; +const max_native_checks = 1_000_000; +const Pending = struct { raw: layout.BBox, display: layout.BBox, segments: u32, subpaths: u32, parent: usize }; + +pub fn classify(allocator: std.mem.Allocator, evidence: *const vector.Evidence, spans: []const layout.TextSpan, page_bbox: layout.BBox, page_geometry: ?geometry.PageGeometry) !Result { + var summary = Summary{ + .status = evidence.status, + .shape_count = @intCast(evidence.shapes.len), + .unknown_count = evidence.unknown_count, + .max_shapes = evidence.limits.max_shapes, + .max_segments_per_path = evidence.limits.max_segments_per_path, + .max_operators = evidence.limits.max_operators, + .operator_count = evidence.operator_count, + .content_bytes = evidence.content_bytes, + .max_content_bytes = evidence.limits.max_content_bytes, + .max_graphics_depth = evidence.limits.max_graphics_depth, + .max_form_invocations = evidence.limits.max_form_invocations, + }; + var pending: [max_shapes]Pending = undefined; + var count: usize = 0; + var native_checks: usize = 0; + for (evidence.shapes) |shape| { + if (shape.clip_state == .outside) { + summary.clipped_out_count += 1; + continue; + } + if (shape.clip_state == .unknown) { + summary.ambiguous_shape_count += 1; + continue; + } + const raw = intersection(shape.bbox, page_bbox) orelse { + summary.clipped_out_count += 1; + continue; + }; + const display = if (page_geometry) |page| geometry.mapBBox(page.pdfToDisplay(.media), raw) catch { + summary.ambiguous_shape_count += 1; + continue; + } else raw; + const width = display.x1 - display.x0; + const height = display.y1 - display.y0; + if (@min(width, height) <= 3 and @max(width, height) >= @max(1, @min(width, height)) * 12) { + summary.ignored_rule_count += 1; + continue; + } + const coverage = nativeCoverage(raw, spans, &native_checks); + if (!coverage.complete) { + if (summary.status != .limited) summary.status = .partial; + summary.ambiguous_shape_count += 1; + continue; + } + if (coverage.fraction >= 0.75) { + summary.native_accounted_count += 1; + continue; + } + if (shape.rectangle_only) { + summary.simple_graphic_count += 1; + continue; + } + if (count == pending.len) { + summary.status = .limited; + summary.ambiguous_shape_count += 1; + continue; + } + pending[count] = .{ .raw = raw, .display = display, .segments = shape.segment_count, .subpaths = shape.subpath_count, .parent = count }; + count += 1; + } + if (native_checks >= max_native_checks and summary.status != .limited) summary.status = .partial; + for (0..count) |i| { + for (i + 1..count) |j| { + const a = pending[i].display; + const b = pending[j].display; + const ha = a.y1 - a.y0; + const hb = b.y1 - b.y0; + const h = @max(ha, hb); + const dy = @abs((a.y0 + a.y1) / 2 - (b.y0 + b.y1) / 2); + const gap = @max(a.x0, b.x0) - @min(a.x1, b.x1); + if (h <= @max(0.1, @min(ha, hb)) * 3 and dy <= h * 0.6 and gap <= h * 0.8) { + const first = find(&pending, i); + const second = find(&pending, j); + pending[second].parent = first; + } + } + } + var candidates: std.ArrayList(Candidate) = .empty; + errdefer candidates.deinit(allocator); + for (0..count) |i| { + if (find(&pending, i) != i) continue; + var raw = pending[i].raw; + var display = pending[i].display; + var shapes: u32 = 0; + var segments: u32 = 0; + var subpaths: u32 = 0; + for (0..count) |j| { + if (find(&pending, j) != i) continue; + raw = unionBox(raw, pending[j].raw); + display = unionBox(display, pending[j].display); + shapes += 1; + segments += pending[j].segments; + subpaths += pending[j].subpaths; + } + const ratio = (display.x1 - display.x0) / @max(0.1, display.y1 - display.y0); + // Several similarly placed contours suggest lettering. Single symbols, + // oversized panels and dense drawings remain ambiguous evidence. + if ((shapes < 3 and subpaths < 3) or segments < @max(10, shapes * 6) or ratio < 1.2 or ratio > 80 or shapes > 128) { + summary.ambiguous_shape_count += shapes; + continue; + } + if (candidates.items.len == summary.max_candidate_regions) { + summary.status = .limited; + summary.ambiguous_shape_count += shapes; + continue; + } + const coverage = nativeCoverage(raw, spans, &native_checks); + if (!coverage.complete) { + if (summary.status != .limited) summary.status = .partial; + summary.ambiguous_shape_count += shapes; + continue; + } + try candidates.append(allocator, .{ + .bbox = raw, + .shape_count = shapes, + .segment_count = segments, + .subpath_count = subpaths, + .native_coverage = coverage.fraction, + }); + } + summary.candidate_region_count = @intCast(candidates.items.len); + if (native_checks >= max_native_checks and summary.status != .limited) summary.status = .partial; + return .{ .allocator = allocator, .summary = summary, .candidates = try candidates.toOwnedSlice(allocator) }; +} + +fn find(items: *[max_shapes]Pending, start: usize) usize { + var index = start; + while (items[index].parent != index) index = items[index].parent; + return index; +} +fn intersection(a: layout.BBox, b: layout.BBox) ?layout.BBox { + const box = layout.BBox{ .x0 = @max(a.x0, b.x0), .y0 = @max(a.y0, b.y0), .x1 = @min(a.x1, b.x1), .y1 = @min(a.y1, b.y1) }; + return if (box.x1 > box.x0 and box.y1 > box.y0) box else null; +} +fn unionBox(a: layout.BBox, b: layout.BBox) layout.BBox { + return .{ .x0 = @min(a.x0, b.x0), .y0 = @min(a.y0, b.y0), .x1 = @max(a.x1, b.x1), .y1 = @max(a.y1, b.y1) }; +} +const Interval = struct { start: f64, end: f64 }; +fn less(_: void, a: f64, b: f64) bool { + return a < b; +} +fn intervalLess(_: void, a: Interval, b: Interval) bool { + return a.start < b.start; +} + +/// Exact rectangle union prevents overlapping native spans inflating coverage. +const Coverage = struct { fraction: f64 = 0, complete: bool = true }; +fn nativeCoverage(box: layout.BBox, spans: []const layout.TextSpan, checks: *usize) Coverage { + var rects: [64]layout.BBox = undefined; + var count: usize = 0; + for (spans) |span| { + if (checks.* >= max_native_checks) return .{ .complete = false }; + checks.* += 1; + if (span.paint.paint_state != .painting or span.paint.clip_state == .outside or span.paint.clip_state == .unknown) continue; + if (std.mem.trim(u8, span.text, " \r\n\t").len == 0) continue; + const visible = if (span.paint.clip_bbox) |clip| intersection(span.bbox, clip) orelse continue else span.bbox; + if (intersection(box, visible)) |rect| { + if (count == rects.len) return .{ .complete = false }; + rects[count] = rect; + count += 1; + } + } + var xs: [128]f64 = undefined; + for (rects[0..count], 0..) |rect, i| { + xs[i * 2] = rect.x0; + xs[i * 2 + 1] = rect.x1; + } + std.mem.sort(f64, xs[0 .. count * 2], {}, less); + var covered: f64 = 0; + if (count > 0) for (xs[0 .. count * 2 - 1], 0..) |x, i| { + const next = xs[i + 1]; + if (next <= x) continue; + var ys: [64]Interval = undefined; + var n: usize = 0; + for (rects[0..count]) |rect| if (rect.x0 <= x and rect.x1 >= next) { + ys[n] = .{ .start = rect.y0, .end = rect.y1 }; + n += 1; + }; + std.mem.sort(Interval, ys[0..n], {}, intervalLess); + var length: f64 = 0; + var end: f64 = -std.math.inf(f64); + for (ys[0..n]) |y| { + length += @max(0, y.end - @max(end, y.start)); + end = @max(end, y.end); + } + covered += (next - x) * length; + }; + return .{ .fraction = @min(1, covered / ((box.x1 - box.x0) * (box.y1 - box.y0))) }; +} + +test "native coverage unions overlap and ignores invisible text" { + const box = layout.BBox{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 10 }; + var span = layout.TextSpan.init(.{ .text = "caption", .bbox = .{ .x0 = 0, .y0 = 0, .x1 = 20, .y1 = 10 }, .paint = .{ .paint_state = .painting, .clip_state = .unclipped } }); + var checks: usize = 0; + try std.testing.expectApproxEqAbs(@as(f64, 0.2), nativeCoverage(box, &.{ span, span }, &checks).fraction, 0.0001); + span.paint.paint_state = .non_painting; + try std.testing.expectEqual(@as(f64, 0), nativeCoverage(box, &.{span}, &checks).fraction); +} + +fn testShapes() [3]vector.Shape { + var shapes: [3]vector.Shape = undefined; + for (&shapes, 0..) |*shape, i| { + const x: f64 = 10 + @as(f64, @floatFromInt(i)) * 7; + shape.* = .{ .bbox = .{ .x0 = x, .y0 = 10, .x1 = x + 6, .y1 = 20 }, .paint_kind = .fill, .rectangle_only = false, .segment_count = 10, .subpath_count = 1, .clip_state = .unclipped, .content_scope = null }; + } + return shapes; +} +fn classificationProbe(allocator: std.mem.Allocator) !void { + var shapes = testShapes(); + const evidence = vector.Evidence{ .allocator = allocator, .shapes = &shapes, .status = .complete, .operator_count = 30, .unknown_count = 0, .limits = .{} }; + var result = try classify(allocator, &evidence, &.{}, .{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 100 }, null); + defer result.deinit(); + try std.testing.expectEqual(@as(usize, 1), result.candidates.len); +} +test "vector classifier candidate ownership survives allocation failures" { + try std.testing.checkAllAllocationFailures(std.testing.allocator, classificationProbe, .{}); +} +test "vector classification respects native coverage graphics and clipping" { + var shapes = testShapes(); + const evidence = vector.Evidence{ .allocator = std.testing.allocator, .shapes = &shapes, .status = .complete, .operator_count = 30, .unknown_count = 0, .limits = .{} }; + const page = layout.BBox{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 100 }; + const native = layout.TextSpan.init(.{ .text = "native", .bbox = page, .paint = .{ .paint_state = .painting, .clip_state = .unclipped } }); + var accounted = try classify(std.testing.allocator, &evidence, &.{native}, page, null); + defer accounted.deinit(); + try std.testing.expectEqual(@as(u32, 3), accounted.summary.native_accounted_count); + try std.testing.expectEqual(@as(usize, 0), accounted.candidates.len); + for (&shapes) |*shape| shape.rectangle_only = true; + var graphics = try classify(std.testing.allocator, &evidence, &.{}, page, null); + defer graphics.deinit(); + try std.testing.expectEqual(@as(u32, 3), graphics.summary.simple_graphic_count); + for (&shapes) |*shape| shape.clip_state = .outside; + var clipped = try classify(std.testing.allocator, &evidence, &.{}, page, null); + defer clipped.deinit(); + try std.testing.expectEqual(@as(u32, 3), clipped.summary.clipped_out_count); +} + +test "partial native clips cannot explain the unpainted portion of a span" { + const box = layout.BBox{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 10 }; + const span = layout.TextSpan.init(.{ .text = "wide native text", .bbox = box, .paint = .{ + .paint_state = .painting, + .clip_state = .partial, + .clip_bbox = .{ .x0 = 0, .y0 = 0, .x1 = 1, .y1 = 10 }, + } }); + var checks: usize = 0; + try std.testing.expectApproxEqAbs(@as(f64, 0.01), nativeCoverage(box, &.{span}, &checks).fraction, 0.00001); +} + +test "native coverage exhaustion abstains instead of publishing zero coverage" { + var shapes = testShapes(); + const page = layout.BBox{ .x0 = 0, .y0 = 0, .x1 = 100, .y1 = 100 }; + const native = layout.TextSpan.init(.{ .text = "native", .bbox = page, .paint = .{ .paint_state = .painting, .clip_state = .unclipped } }); + var spans: [65]layout.TextSpan = @splat(native); + const evidence = vector.Evidence{ .allocator = std.testing.allocator, .shapes = &shapes, .status = .complete, .operator_count = 30, .unknown_count = 0, .limits = .{} }; + var result = try classify(std.testing.allocator, &evidence, &spans, page, null); + defer result.deinit(); + try std.testing.expectEqual(.partial, result.summary.status); + try std.testing.expectEqual(@as(u32, 3), result.summary.ambiguous_shape_count); + try std.testing.expectEqual(@as(usize, 0), result.candidates.len); +} From 075d9250a88e7468a883b5c27175de3990de4ac7 Mon Sep 17 00:00:00 2001 From: Tom Main Date: Sun, 11 Oct 2026 16:24:23 -0400 Subject: [PATCH 2/2] Sync committed coordinate and vector contract documentation --- README.md | 6 +++--- docs/host-adapters.md | 13 +++++++------ docs/output-schema.md | 11 ++++++----- docs/parser-host-implementation-plan.md | 6 +++--- 4 files changed, 19 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index d54eb41..79b81da 100644 --- a/README.md +++ b/README.md @@ -672,9 +672,9 @@ OCR, table, formula, and form spans with typed provenance. The versioned JSON, artifact JSONL, and streaming JSONL schema is currently `0.19.0` in local development, with parser build identity `0.4.0-dev` after the `v0.3.0` release. Schema `0.17.0` landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6) -at `9cac608`, with all six CI checks green. Schema `0.18` is locally checked and -uncommitted; remote CI and merge remain separate. The `0.19` vector-routing -continuation is locally checked and uncommitted. Its 23 outlined-text and +at `9cac608`, with all six CI checks green. Schemas `0.18` and `0.19` are locally +checked and committed on `codex/coordinate-vector-ocr`; remote CI and merge +remain separate. The `0.19` vector-routing continuation's 23 outlined-text and negative-control fixtures, 13 real CLI/SDK tests and both JSONL readers are qualified in the [implementation evidence](docs/parser-host-implementation-plan.md#bounded-vector-routing-019-locally-checked). Every emitted record carries a `provenance` envelope with document and diff --git a/docs/host-adapters.md b/docs/host-adapters.md index fcd2aed..dbedb06 100644 --- a/docs/host-adapters.md +++ b/docs/host-adapters.md @@ -98,7 +98,8 @@ These forward `--ocr-max-image-pixels`, `--ocr-max-image-bytes`, `--ocr-max-regions` and `--ocr-max-attempts`. Caps bound accepted rasters and per-page invocation counts; child RSS and temporary disk growth are outside these budgets. Crop records retain actual PDF-space bounds, accepted PNG SHA-256 -and byte length. Rotation is explicitly unsupported for OCR crops. +and byte length. Declared quarter-turn rotations use the page's raster transform; +unsupported rotations retain explicit refusal outcomes. The result retains the manifest, raw JSONL, all protocol records, spans, blocks and parser chunks. Its `sourceText` is exact block text in artifact order joined @@ -307,10 +308,10 @@ request schema version. Host adoption remains a separate step. Schema `0.17` landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6), merge commit `9cac608`, with all six CI checks green. The coordinate and rotated -OCR continuation is locally checked, uncommitted schema `0.18`. The `0.17` -CI result does not qualify this continuation. -Schema `0.19` adds bounded vector candidates; it is locally checked and -uncommitted. The new vector suite passes all 13 tests with no skips, including +OCR continuation is locally checked schema `0.18`. The `0.17` CI result does +not qualify this continuation. Schemas `0.18` and `0.19` are committed on +`codex/coordinate-vector-ocr`. Schema `0.19` adds bounded vector candidates. +The new vector suite passes all 13 tests with no skips, including actual OCR in both SDK formats and disjoint output in a reused bitmap crop. Remote CI, merge and host adoption remain separate. @@ -318,7 +319,7 @@ Run the local adapter regressions and actual CLI composition test: ```sh zig build --summary all -node --test integrations/javascript/*.test.mjs +node --test --test-concurrency=1 integrations/javascript/*.test.mjs zig build test --summary all ``` diff --git a/docs/output-schema.md b/docs/output-schema.md index f57ce00..2a18446 100644 --- a/docs/output-schema.md +++ b/docs/output-schema.md @@ -6,7 +6,8 @@ should depend on the records documented here. ## Version -Current development schema version: `0.19.0` (locally checked, uncommitted). +Current development schema version: `0.19.0` (locally checked, committed on +`codex/coordinate-vector-ocr`). Current parser build identity: `0.4.0-dev`. The `v0.3.0` release emitted schema `0.11.0`; the development suffix keeps post-release schema `0.19.0` artifacts @@ -38,7 +39,7 @@ Schema `0.19.0` adds required nullable vector evidence on page and region route traces. Bounded path collection proposes geometric OCR candidates without changing the raw-coordinate or source-text contract. Readers must explicitly accept `0.19.x`; formula children must echo it. This unit is locally checked and -uncommitted. Its [qualification](parser-host-implementation-plan.md#bounded-vector-routing-019-locally-checked) +committed. Its [qualification](parser-host-implementation-plan.md#bounded-vector-routing-019-locally-checked) is separate from earlier local results and remote CI. Schema `0.18.0` adds nullable page geometry, OCR raster transforms and formula @@ -191,9 +192,9 @@ open before page artifacts exist, so they include the same artifact slots with such as native text, span modeling, layout reconstruction, complexity routing, reconciliation, table reconstruction, form fields, OCR adapter support, formula routing, specialist protocol support, debug assets, and streaming. Formula -recognition remains false because the configured formula adapter currently -operates on native span/block context without a rendered visual crop. It is a -protocol and lifecycle integration, not yet evidence of visual formula quality. +recognition is true to advertise the configured crop-backed formula adapter. +This capability flag does not establish visual formula accuracy or imply that +an enabled recognizer is bundled. ### `span` diff --git a/docs/parser-host-implementation-plan.md b/docs/parser-host-implementation-plan.md index b98cacc..a1f8a88 100644 --- a/docs/parser-host-implementation-plan.md +++ b/docs/parser-host-implementation-plan.md @@ -1,6 +1,6 @@ # Parser, Research consumer, and Editorial consumer implementation plan -Status: phases 0/1, reconciled table links, bounded streaming and recovery assessments landed in [PR #5](https://github.com/Tom-R-Main/pdf-parser/pull/5), merge commit `0dd534d`. Paint/clipping, bounded regional OCR and text fidelity landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6), merge commit `9cac608`, with all six CI checks green for schema 0.17. Coordinate transforms and rotated OCR are locally checked, uncommitted schema 0.18. Bounded vector routing is locally checked, uncommitted schema 0.19; remote CI and merge remain separate. Host adoption remains pending, 2026-10-11. All writes are restricted to pdf-parser: planned host edits below remain adoption instructions. Based on the [local reference review](local-reference-review-2026-10-09.md) and [host use-case review](host-usecase-reference-review-2026-10-10.md). Richer hierarchy/figures and host rollout remain planned. +Status: phases 0/1, reconciled table links, bounded streaming and recovery assessments landed in [PR #5](https://github.com/Tom-R-Main/pdf-parser/pull/5), merge commit `0dd534d`. Paint/clipping, bounded regional OCR and text fidelity landed in [PR #6](https://github.com/Tom-R-Main/pdf-parser/pull/6), merge commit `9cac608`, with all six CI checks green for schema 0.17. Coordinate transforms and rotated OCR (schema 0.18) and bounded vector routing (schema 0.19) are locally checked and committed on `codex/coordinate-vector-ocr`; remote CI and merge remain separate. Host adoption remains pending, 2026-10-11. All writes are restricted to pdf-parser: planned host edits below remain adoption instructions. Based on the [local reference review](local-reference-review-2026-10-09.md) and [host use-case review](host-usecase-reference-review-2026-10-10.md). Richer hierarchy/figures and host rollout remain planned. ## Delivery order and ownership @@ -360,8 +360,8 @@ Whole-Python-process shutdown remains unqualified, and `memcheck_proven` remains false. These checks do not establish a whole-process memory bound. Evidence and checked binaries are retained under ignored -`benchmark/eval/outputs/vector-routing-2026-10-11/`. This unit remains -uncommitted; no remote CI or merge is claimed. +`benchmark/eval/outputs/vector-routing-2026-10-11/`. This unit is committed on +`codex/coordinate-vector-ocr`; no remote CI or merge is claimed here. Table topology and reading order need new held-out suites; the earlier reading-order holdout must not be retuned or promoted. Host adoption remains a