Repository navigation
Conversation
The empty compact sketch check now uses a mask that ignores the ordered flag, so it accepts empty images written by C++ before 3.3.0 and Java before 1.0.0, which did not always set it. Bytes 3 and 4 must be zero and reserved flag bits 0, 6 and 7 must be clear. It also accepts the legacy 24-byte empty form (preamble longs 3) that Java wrote before 1.0.0 for sketches with p < 1. That form must have no retained entries and carry the expected seed hash. CompactThetaSketch.heapify now applies the same check as wrap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EmptyCompactSketchdecides whether the first 8 bytes of a serial version 3 compact image are a valid empty sketch, using a mask (which bits are examined) and a test (their required values). This PR corrects both so that every empty image written by an open-source Java or C++ release is accepted, while mangled images are still rejected. The matching C++ change is apache/datasketches-cpp#543, which uses the same mask and test values.0X00_00_EB_00_00_FF_FF_FFL0X00_00_EF_FF_FF_FF_FF_FFL0X00_00_0A_00_00_03_03_01L0X00_00_0E_00_00_03_03_01L0X00_00_0E_00_00_03_03_03L{1, 3, 3, 0, 0, 0x1E, 0, 0}The mask and test now satisfy two rules, checked by a unit test: the test lies within the mask, and our own array passes.
Changes
0x0E). The current mask onmain(from a9954b9, not in any release) requires it.wraphas rejected these since 2019. They are now accepted if they have no retained entries and carry the expected seed hash, and are read asEmptyCompactSketchwith theta = 1.0.CompactThetaSketch.heapifynow applies the same check aswrap. Before, it accepted any image with the empty flag set.No serial version change: the writer is unchanged. Serial version 4 is not affected, because empty sketches are never written compressed.
Testing
EmptyTestcases check the two rules, accept 5 historical images and reject 12 mangled ones, throughheapify,wrap(MemorySegment)andwrap(byte[]).heapifyand bothwrapoverloads: 2 from sketches-core 0.9.0 (p = 0.5, ordered and unordered), and 60 written by C++ 1.0.0-incubating, 2.0.0-incubating, 3.0.0, 3.2.0, 3.3.0 and current master (10 empty cases each: update compact, union, intersection and A-not-B, with p = 1 and 0.5, ordered and unordered).🤖 Generated with Claude Code