Skip to content

Theta: fix EmptyCompactSketch preamble check for historical images - #776

Open
leerho wants to merge 1 commit into
mainfrom
theta-empty-mask
Open

leerho wants to merge 1 commit into
mainfrom
theta-empty-mask

Conversation

@leerho

@leerho leerho commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Summary

EmptyCompactSketch decides whether the first 8 bytes of a serial version 3 compact image are a valid empty sketch, using a mask (which bits are examined) and a test (their required values). This PR corrects both so that every empty image written by an open-source Java or C++ release is accepted, while mangled images are still rejected. The matching C++ change is apache/datasketches-cpp#543, which uses the same mask and test values.

9.0.0 This PR
Mask 0X00_00_EB_00_00_FF_FF_FFL 0X00_00_EF_FF_FF_FF_FF_FFL
Test 0X00_00_0A_00_00_03_03_01L 0X00_00_0E_00_00_03_03_01L
Legacy test (preamble longs 3) none 0X00_00_0E_00_00_03_03_03L
Array written {1, 3, 3, 0, 0, 0x1E, 0, 0} unchanged

The mask and test now satisfy two rules, checked by a unit test: the test lies within the mask, and our own array passes.

Changes

  • Ordered flag ignored. C++ before 3.3.0 and Java before 1.0.0 wrote empty images without it (0x0E). The current mask on main (from a9954b9, not in any release) requires it.
  • Bytes 3 and 4 must be zero. They have been "not used by compact" since 0.1.0. 9.0.0 ignored them.
  • Reserved flag bits 0, 6 and 7 must be clear. Any future use of them requires a new serial version.
  • Empty flag checked. Both callers had already confirmed it, so this only makes the test complete.
  • Legacy 24-byte form accepted. Before 1.0.0, Java wrote empty sketches with p < 1 with preamble longs 3 and theta = p. wrap has rejected these since 2019. They are now accepted if they have no retained entries and carry the expected seed hash, and are read as EmptyCompactSketch with theta = 1.0.
  • CompactThetaSketch.heapify now applies the same check as wrap. Before, it accepted any image with the empty flag set.

No serial version change: the writer is unchanged. Serial version 4 is not affected, because empty sketches are never written compressed.

Testing

  • Full suite passes (2290 tests).
  • New EmptyTest cases check the two rules, accept 5 historical images and reject 12 mangled ones, through heapify, wrap(MemorySegment) and wrap(byte[]).
  • Real images read as empty by heapify and both wrap overloads: 2 from sketches-core 0.9.0 (p = 0.5, ordered and unordered), and 60 written by C++ 1.0.0-incubating, 2.0.0-incubating, 3.0.0, 3.2.0, 3.3.0 and current master (10 empty cases each: update compact, union, intersection and A-not-B, with p = 1 and 0.5, ordered and unordered).

🤖 Generated with Claude Code

The empty compact sketch check now uses a mask that ignores the ordered
flag, so it accepts empty images written by C++ before 3.3.0 and Java
before 1.0.0, which did not always set it. Bytes 3 and 4 must be zero
and reserved flag bits 0, 6 and 7 must be clear.

It also accepts the legacy 24-byte empty form (preamble longs 3) that
Java wrote before 1.0.0 for sketches with p < 1. That form must have no
retained entries and carry the expected seed hash.

CompactThetaSketch.heapify now applies the same check as wrap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant