Skip to content

Theta: validate the preamble of empty compact sketches as Java does - #543

Open
leerho wants to merge 1 commit into
masterfrom
theta-empty-mask
Open

leerho wants to merge 1 commit into
masterfrom
theta-empty-mask

Conversation

@leerho

@leerho leerho commented Oct 8, 2026

Copy link
Copy Markdown
Member

Summary

C++ accepted any serial version 3 compact image with the empty flag set, without checking the rest of the preamble. This PR applies the same check as Java EmptyCompactSketch (apache/datasketches-java#776), using the same mask and test values, so both languages accept and reject the same empty images.

Value
Mask 0x0000EFFFFFFFFFFF
Test (preamble longs 1) 0x00000E0000030301
Legacy test (preamble longs 3) 0x00000E0000030303

Changes

  • New compact_theta_sketch_parser::check_empty_v3, used by parse (byte deserialize and wrap) and by stream deserialize_v3.
  • Checked: preamble longs, serial version, sketch type, bytes 3 and 4 zero, read-only, empty and compact set, single-item and reserved bits 0, 6 and 7 clear.
  • Ordered flag ignored. C++ before 3.3.0 and Java before 1.0.0 wrote empty images without it.
  • Seed hash ignored in the 8-byte form. Writers have used both 0 and the real seed hash.
  • Legacy 24-byte form accepted. Before 1.0.0, Java wrote empty sketches with p < 1 with preamble longs 3. These must have no entries and carry the expected seed hash. C++ never wrote this form.
  • Stream position fixed. For a legacy image, stream deserialize_v3 now reads all 24 bytes instead of stopping after 8.

Behavior change for users: empty images with non-zero bytes 3 and 4, read-only or compact missing, single-item or a reserved bit set, preamble longs 2, or a malformed legacy form are now rejected. No C++ or Java release I tested writes any of these.

Testing

  • Full suite passes (17/17).
  • Three new test cases mirror the Java ones: the mask and test rules, 5 historical images accepted and 12 mangled images rejected, each through stream deserialize, byte deserialize and wrap.
  • Real images read as empty through all three entry points: 60 written by C++ 1.0.0-incubating, 2.0.0-incubating, 3.0.0, 3.2.0, 3.3.0 and current master, and 2 from Java sketches-core 0.9.0.

🤖 Generated with Claude Code

Empty serial version 3 compact images were accepted on the empty flag
alone. They are now checked with the same mask and test values as Java
EmptyCompactSketch: preamble longs, serial version, sketch type, bytes
3 and 4 zero, and the read-only, empty, compact, single-item and
reserved flag bits. The ordered flag is ignored, since C++ before 3.3.0
and Java before 1.0.0 wrote empty images without it.

The legacy 24-byte empty form (preamble longs 3) written by Java before
1.0.0 for sketches with p < 1 is accepted if it has no entries and
carries the expected seed hash. The stream deserializer now reads the
whole legacy image instead of stopping after the first 8 bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 37711570993

Coverage increased (+0.03%) to 82.409%

Details

  • Coverage increased (+0.03%) from the base build.
  • Patch coverage: 26 of 26 lines across 2 files are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 21454
Covered Lines: 17680
Line Coverage: 82.41%
Coverage Strength: 1371453.49 hits per line

💛 - Coveralls

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants