Repository navigation
[dfast] Use ZSTD_selectAddr() for prefix checks in dictMatchState - #4866
Open
jeffchen006 wants to merge 1 commit into
Open
jeffchen006 wants to merge 1 commit into
jeffchen006 wants to merge 1 commit into
Conversation
ZSTD_compressBlock_doubleFast_dictMatchState_generic() tests matchIndex >= prefixLowestIndex before it reads the prefix candidate. When the prefix is small, which is the usual case for small inputs with an attached dictionary, this branch is hard to predict. Apply the method of meta#4144 / meta#4165 (noDict) to the three prefix checks (long at ip, short at ip, long at ip+1): read through ZSTD_selectAddr(), which returns a local dummy buffer for an invalid index, and test the index only after the bytes compare equal. The index test still decides the result, so the compressed output is unchanged. silesia.tar compressed in independent 1-16 KB blocks with a 16 KB raw dictionary, AMD EPYC 9654, median of 10-12 alternating pairs: clang 14: levels 3-4 +14% to +17% gcc 11: levels 3-4 +2.6% to +4.0% Branch misses drop by 35% (gcc) and 65% (clang). Streaming with an attached dictionary and unknown size, on 1 MB inputs, is +2.4% to +4.7%. Without a dictionary, or with blocks > 16 KB, this code does not run and the results are neutral. Code size: text +1320 bytes (gcc), +2944 bytes (clang). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jeffchen006
marked this pull request as ready for review
October 27, 2026 00:02
This was referenced Nov 4, 2026
zcsizmadia
added a commit
to zcsizmadia/zstd-native
that referenced
this pull request
Nov 6, 2026
…#71) * profile_encode: streaming-with-flushes and dictionary modes (#62) New modes, with the same calls on both libraries: - native-flush / c-flush: one frame through StreamEncoder / ZSTD_compressStream2 with a flush (ZSTD_e_flush) every `chunk` bytes (default 4096); ZSTD_BENCH_WLOG sets the window log. Once the input buffer wraps, blocks go through the extDict match finders. - native-dict / c-dict: frames of `chunk` bytes (default 4096) with a prepared dictionary (EncoderDictionary / ZSTD_createCDict + ZSTD_CCtx_refCDict + ZSTD_compress2), from up to 4 MiB of the dataset. Small inputs attach the dictionary (dictMatchState), larger ones copy its tables (extDict). ZSTD_BENCH_DICT=raw:n | trained:n | <file>. A ZSTD_BENCH_CORPUS file named as the dataset is now read alone (quick start, clean perf stat counts). Baseline, main b33e172 against libzstd 1.5.7 (EPYC 7543, CPU 8, medians of 9 runs, silesia dickens/samba/xml): libzstd was faster by - flush every 4/16/64 KiB: L1 0-6%, L2 -4..+4.5%, L3 2-10%, L4 1.5-9%; - dictionary, 1-4 KiB inputs: L1 12-18%, L2 9-15%, L3 6-11%, L4 6-11%; 16 KiB inputs: zstd-native faster at L1/L2 (2-7%), libzstd at L3/L4 (5-11%). * dfast extDict: out of line, BMI2, cmov candidate checks (meta/zstd#4764) (#62) The extDict loop (streaming once the input buffer wraps, copied dictionaries) was inlined into build_seq_store with every block invariant spilled. Now: - out of line, with a BMI2 build (shrx hashes); - the table candidates' segment and validity selected with cmovs, the index tested only once the bytes match (ZSTD_match8Found_cmov / match4Found_cmov of #4764, through the new fast::cand_match / select_addr / then_valid; an empty asm! keeps the branch order), used from windowLog >= max(chainLog, 16): with smaller windows nearly every candidate is stale, the branch predicts well and is 1-8% faster. Same parse as libzstd 1.5.7. #4764's pipelined loop changes the parse (its own PR: -0.15% bytes), so it is not ported; loading the next position's entries ahead within 1.5.7's parse, and prefetching them 2-4 positions ahead, measured slower (more instructions; the loop is not table-miss bound). Flush every 4/16/64 KiB (dickens, samba, xml), against main: L3 +3.7..+5.2%, L4 +2.6..+8.5%; 32 KiB window L3 +1.6..+1.8%, L4 +2.9..+3.6%. L1/L2 and one-shot unchanged (control: +-0.8%). * dfast dictMatchState: cmov prefix checks (meta/zstd#4866), out of line (#62) The three prefix candidate checks (long and short at ip, long at ip+1) read through ZSTD_selectAddr and test the index only once the bytes match: with an attached dictionary the prefix is short, and whether a candidate is in it is an unpredictable branch. The loop also moves out of build_seq_store (0.5-1.5% faster than inlined); a BMI2 build measured 1.5-3% slower here and is not used. Dictionary, 1-16 KiB inputs, against the previous commit: raw 16 KiB dictionary L3 +0.5..+6.2%, L4 +1.5..+7.2%; trained 64 KiB dictionary -1.0..+2.8% (the -1.0% is xml 16 KiB L4; control noise +-0.8%). * fast dictMatchState: ZSTD_match4Found_cmov, BMI2 (#62) libzstd 1.5.7 checks the prefix candidate of the fast dictMatchState loop with ZSTD_match4Found_cmov; the port used a branch on the index first, mispredicted with the short prefix of small inputs. Now the same cmov check, and a BMI2 build of the loop. Dictionary, inputs that attach (L1/L2 up to 8 KiB), against the previous commit: L1 +6.0..+13.4%, L2 +4.0..+7.3% (branch misses -38% at L1); 16 KiB inputs (copied dictionary) -0.6..+0.2%. * fast extDict: cmov candidate checks for windows up to 64 KiB (#62) When the valid indexes span at most 64 KiB (small windows, small copied dictionaries), many table entries are stale: the candidate's segment and validity are selected with cmovs (meta/zstd#4764) in a BMI2 build. Larger windows keep libzstd's branches, in their own out-of-line function (code placement: inside one function with the cmov path they measured 1-2% slower with identical instructions). Against the previous commit: flush every 4 KiB with a 32 KiB window L1 +11%, L2 +8%; 64 KiB window L1 +3..+8%; 16 KiB inputs with a 16 KiB dictionary (copied) L1 +8%, L2 +7%. 128 KiB windows and up, and a copied 64 KiB dictionary: -0.4..+1.0% (control +-0.5%). * stream_encode: frequent flushes at the fast levels (#62) A flush every 1-16 KiB at levels -3..4 with 1 KiB to 128 KiB windows and the default ones (past them: 3 MB of input), small output buffers: nearly every block in extDict mode, against libzstd's bytes. * CLAUDE.md: the new unsafe and asm! locations (#62) The run-time BMI2 dispatch of the fast/dfast extDict and fast dictMatchState loops, and then_valid's empty asm! barrier.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the dictionary follow-up suggested at the end of #4165:
ZSTD_compressBlock_doubleFast_dictMatchState_generic()testsmatchIndex >= prefixLowestIndexbefore it reads a prefix candidate. With an attached dictionary, the input is usually small and the prefix is short, so many candidate indexes come from before the prefix. The branch is then hard to predict.This PR applies the method of #4144 / #4165 to the three prefix checks in this function: the long match at
ip, the short match atip, and the long match atip+1.ZSTD_selectAddr(), which returns a localdummy[]buffer when the index is invalid. This is the samedummy[]as in the noDict function.The index test still decides the result, so the compressed output is byte-identical.
Unlike #4165, this does not need a separate branch-based variant for long inputs. With an attached dictionary, this function runs only while the dictionary is in the window, and long streamed inputs are faster as well (see below).
Results
silesia.tarcompressed as independent blocks (zstd -b<level> -i2 -B<size> -D <dict>), with a 16 KB raw-content dictionary: the first 16,384 bytes ofdickens(head -c 16384 dickens > raw.dict). Blocks of 16 KB or less attach the dictionary, so they usedictMatchState. Each cell is the median of 10 alternatingdev/PR pairs (12 for the 16 KB rows) on one pinned core. "Change" is the median of the per-pair ratios.devMB/sCompressed sizes are identical to
devin every cell. A/A on the same core (-B16K -D, level 3): gcc +0.33%, clang +0.41%. Decompression is not changed, and all its rows are within −0.4% to +0.7%.The results are similar with trained dictionaries (64 KB and 112 KB, trained on 4 KB slices of Silesia), level 3:
Mechanism (
perf stat, per pass over 12 × 1 MB Silesia files, level 3, 16 KB chunks withZSTD_compress_usingCDict()):Long inputs with an attached dictionary. #4165 found that the
cmovmethod makes long noDict inputs slower. Here, the matching case is a stream with unknown size: the CDict stays attached, and the prefix grows up to the window size. I measured it withZSTD_compressStream2(),ZSTD_CCtx_refCDict(), 128 KBZSTD_e_continuesteps, the same 16 KB dictionary, and the 1 MB prefixes of the 12 Silesia files.perfconfirms thatZSTD_compressBlock_doubleFast_dictMatchState_*takes 79% of the time. 12 pairs:devMB/sA/A in this mode: gcc +0.17% (6/12), clang +0.34% (9/12).
Unchanged paths (this function is not used):
silesia.tar, no dictionary, level 3silesia.tar, no dictionary, level 4These rows are within the A/A spread.
Cost
Code size.
sizetext:zstdCLIzstd_double_fast.o,-O3zstd_double_fast.o,-O2Adversarial input. The code compares 8 or 4 bytes from
dummy[]with the input when the index is invalid. An input made of thedummy[]byte pattern therefore causes many false-equal reads, and every false-equal read costs an index test. On such a crafted 300 KB input (-B4K, level 3), the PR is −1.6% (gcc) and −4.4% (clang).noDicthas the same property.Correctness
ZSTD_selectAddr(idx, low, match, dummy)returnsmatchif and only ifidx >= low. The two operands of&&have no side effects, so the swapped order gives the same boolean. For a valid index, the read address is the same as before. For an invalid index, the result is false whateverdummy[]contains.ZSTD_selectAddr()uses>=, but the check uses>. Atidx == prefixLowestIndex, the read goes toprefixLowest, which is valid memory, andidx > prefixLowestIndexis false. The dictionary branch is reached under the same condition as before (dictTagsMatchS && idx <= prefixLowestIndex).MEM_read64, and it never reads below the prefix.minMatch4–7,wlog10–17,hlog/clog), chunk sizes from 1 B to 128 KB, attach modes 0/1/2, CDict andloadDictionary, andcompress2and streaming with and without a pledged size. The builds were gcc, clang,-m32and the forced C fallback ofZSTD_selectAddr().idx == prefixLowestIndexand dummy bytes equal to the input.fuzzer -T180s,zstreamtest -T180s, an ASan+UBSanfuzzer -T120s, andplayTests.sh(also with--test-large-data) all passed. ASan+UBSan andDEBUGLEVEL=1library builds passed 66,168 configurations each.Method
make zstdandmake -C lib libzstd.awith default flags (-O3) and no-march.silesia.tar, 211,957,760 bytes, SHA-2560b5ec70c8048321525f820bc1d76429840bd22ee2c7a116175253a0fec2c724d.head -c 16384 dickens > raw.dict(SHA-256626287fc08506a3f5c970220e11a187237abd7a24c86e907699a781b1e6ef88d). The trained dictionaries were made withzstd --trainon 4 KB slices of Silesia, so they favor this data.zstd -b<level> -i2 -B<size> -D <dict> silesia.tar. Each pair runsdevand the PR back to back, and the order alternates between pairs.perfrows: a small C program, statically linked with eachlibzstd.a. In stream mode it usesZSTD_CCtx_refCDict()+ZSTD_compressStream2()with 128 KBZSTD_e_continuesteps, thenZSTD_e_end. In chunk mode it callsZSTD_compress_usingCDict()on each 16 KB chunk. Each run does 1 second of complete passes. The program checks thatdevand the PR produce the same total compressed size. I can add it to the PR if it helps.Limits
ZSTD_selectAddr()is a plain ternary, so the effect depends on whether the compiler emitscsel. Improve compression speed on small blocks #4165 saw a good result on M1 for noDict.dfastdictMatchStatepath (levels 3–4 with an attached dictionary) is changed. TheextDictpath and thefastdictMatchStatepath are not changed.Checklist
A gain in one configuration is not a general gain: see the Performance section of CONTRIBUTING.md.
Tools used
Claude Code (Claude Opus) found this optimization, wrote the patch, ran the correctness checks (byte-identity harnesses, fuzzer, zstreamtest, ASan/UBSan, playTests) and the benchmarks, and drafted this description. A separate Claude Code agent reviewed the patch independently and repeated the measurements. I reviewed the diff and the benchmark data before submitting.