Repository navigation
perf(iluvatar): batch FlashAttnWithKvcache QK reductions - #1026
superme1one wants to merge 7 commits into
Conversation
| [length - i for i in range(batch)], dtype=torch.int32, device=device | ||
| ) | ||
| out = torch.empty_like(q) | ||
| infini.ops.paged_attention_infinilm( |
There was a problem hiding this comment.
This op is deprecated, plz don't use it anymore, you could use flash_attn_with_kvcache instead.
There was a problem hiding this comment.
Updated the tests to call infini.ops.flash_attn_with_kvcache with native implementation 0. The deprecated operator is no longer used by this PR’s tests. All 46 affected native cases passed on TG-V200 without skips.
| @@ -0,0 +1,56 @@ | |||
| import infini.ops | |||
There was a problem hiding this comment.
plz move this test to test_{ops}.py file.
There was a problem hiding this comment.
Moved the regressions into the existing tests/test_flash_attn_with_kvcache.py and removed tests/test_iluvatar_decode_reduction.py. It now contains 24 smoke-marked Python no-workspace/fallback cases and 21 C++ workspace cases for the optimized path. Together with the existing paged regression, all 46 passed on TG-V200. The C++ tests use the existing SDK/compiler test helpers; the command and environment requirements are in the PR description.
| @@ -0,0 +1,91 @@ | |||
| # Iluvatar Decode QK reduction | |||
There was a problem hiding this comment.
This doc is not necessary.
There was a problem hiding this comment.
Removed docs/performance/iluvatar-decode-qk-reduction.md from the PR. The updated measurement method and per-configuration results are now in the PR description.
| bool iluvatar, bool bf16, bool int32, std::size_t batch, std::size_t heads, | ||
| std::size_t kv, std::size_t dim, std::size_t blocks, std::size_t block, | ||
| int splits, bool alibi, std::ptrdiff_t qs, std::ptrdiff_t qh, | ||
| std::ptrdiff_t os, std::ptrdiff_t oh, std::ptrdiff_t kb, std::ptrdiff_t ks, | ||
| std::ptrdiff_t kh, std::ptrdiff_t vb, std::ptrdiff_t vs, | ||
| std::ptrdiff_t vh) { |
There was a problem hiding this comment.
This input list is too long, could we use struct to reduce input number here?
There was a problem hiding this comment.
Replaced the positional eligibility arguments with DecodeQkConfig, which groups type, shape, stride and alignment metadata. Kernel inputs are passed through DecodeQkParams. Added host coverage for the eligibility boundaries and split selection.
| k_cache_slot_stride_, k_cache_head_stride_, | ||
| v_cache_block_stride_, v_cache_slot_stride_, | ||
| v_cache_head_stride_, num_splits); | ||
| const auto launch_split = [&](auto batched) { |
There was a problem hiding this comment.
This op is deprecated link, we don't expect any change on it.
If you need this kernel, please migrate this kernel to this folder
There was a problem hiding this comment.
Migrated the optimized split/combine kernels to src/native/cuda/iluvatar/ops/flash_attn_with_kvcache/ and wired them into Operator<FlashAttnWithKvcache, kIluvatar>. All changes to paged_attention_infinilm have been removed from the PR diff. The new path builds with CoreX and passed the 46 native tests through the maintained interface; its existing fallback is unchanged.
| return iluvatar && bf16 && int32 && (batch == 1 || batch == 2) && | ||
| heads == 32 && kv == 2 && dim == 128 && blocks == 1 && block == 4096 && | ||
| splits == 4 && !alibi && qs == 4096 && qh == 128 && os == 4096 && | ||
| oh == 128 && kb == 1048576 && ks == 128 && kh == 524288 && | ||
| vb == 1048576 && vs == 128 && vh == 524288; |
There was a problem hiding this comment.
It seems that our change take effect for one specifice case, could it improve performance for other case?
There was a problem hiding this comment.
Expanded beyond the original single layout: B=1/2/4/8/16, Hq=8/16/32/64/128, Hkv=1/2/4/8/16, page sizes 256–4096, multiple pages, 4/8 splits, MHA/GQA and padded query/output rows. With explicit workspace through the public C++ interface, the 17 medium/long-cache configurations improved in both same-card role orders by 1.57–2.58x. The two 7-token cases were within 1% of baseline in the averaged measurements. The PR description includes every configuration and latency. All 19 configurations produced bit-identical BF16 outputs across the official and candidate libraries in both role orders. D=128 remains the optimized head dimension, with D64/FP16/B17 fallback coverage. Please take another look; the upstream workflows still need maintainer approval.
Summary
src/native/cuda/iluvatar/ops/flash_attn_with_kvcache/and call it throughflash_attn_with_kvcache. The PR no longer changes the deprecatedPagedAttentionInfinilmimplementation.DecodeQkConfig; pass kernel inputs inDecodeQkParams.tests/test_flash_attn_with_kvcache.pyand remove the standalone performance report from the repository.Motivation
The original PR optimized only B=1/2, Hq=32, Hkv=2 and a single 4096-token cache page through a deprecated operator. This revision uses the maintained interface and measures additional configurations on TG-V200. The per-operator results below compare this revision with the official baseline; earlier combined-model results remain in companion InfiniLM #610.
Type of Change
perf— performance improvementrefactor— migrate the implementation to the maintained operatorPlatforms Affected
WITH_ILUVATAR)Smoke Test Result
Validated on 2026-11-16 on TG-V200 GPU7. Both official
masterf52af91and candidated30e6a5were built from clean source archives in separate directories, using the same CoreX installation and InfiniRT SDK. Both builds use Release,WITH_CPU=ON,WITH_ILUVATAR=ON,WITH_TORCH=OFF,GENERATE_PYTHON_BINDINGS=ON, andINFINI_OPS_OPS=flash_attn_with_kvcache. The candidate host eligibility-policy test passed.The same candidate test module was run against each native library. The C++ workspace tests use the compiler/prefix helpers in
tests/test_cpp_api.py;INFINI_OPS_INSTALL_PREFIXselects the respective isolated installation, with the sameINFINI_RT_ROOTand CoreX include directory inCXXFLAGS.The candidate passes all 21 explicit-workspace cases, 24 no-workspace/fallback cases, and the existing paged regression. Coverage includes B=1/2/4/8/16, Hq=8/16/32/64/128, Hkv=1/2/4/8/16, page sizes 256–4096, 4/8 splits, reordered physical pages, per-sequence lengths, padded query/output rows, zero-length sequences, and FP16/D64/B17 fallbacks. The independent reference uses PyTorch FP32 attention with BF16/FP16 output (
rtol=2e-2,atol=2e-3).The official baseline fails only the two added workspace tests containing zero-length sequences: lengths
(1, 0)and(0,)(case19andcase20). It returns incorrect values for the zero-length rows. Both pass with the candidate's neutral-partial and zero-output handling. These two boundary cases are excluded from the performance table; all 19 timed configurations pass the reference check for both libraries.The 30 skips are existing backend/parameter exclusions: Ascend (6), Moore (9), NVIDIA MLA (4), non-paged Iluvatar cases (10), and scalar cache lengths unsupported by the Iluvatar native provider (1).
Test Results on Supported Platforms
Benchmark / Performance Impact
Remeasured on 2026-11-16, TG-V200 GPU7, after merging current official
master:f52af91a83f19610b16120ab1be4f5c884c02c1c.d30e6a5c3fbd0090d4bb9886162cfaf753f0b279; the merge leaves the Iluvatar fast-path implementation unchanged.OMP_NUM_THREADS=1. Same device, CoreX toolchain, Release build options, InfiniRT SDK, input seeds, tensor shapes and timing method for both libraries.4d5c5356cff8a8f652fa1f9c0eb192b1b61be988792d8482f3018c6bad742553; candidatebf3bd163fddb6697438b4dafcf53165bf78a810eb529bfcab9ebf01eb400f665. Loaded library paths were checked in every benchmark process.Inputs, output and the 8-split workspace are preallocated. Both roles invoke the public C++
FlashAttnWithKvcache::Callwith the same explicit workspace and stream. Each process warms up 20 calls and measures five sets of 100 calls using CUDA events. Three serial official → candidate → candidate → official blocks give six processes and 30 timing samples per role/configuration. Each table entry is the median of six per-process medians. Cache lengths decrease by the batch-row index; the table shows the first row's length. GPU status was recorded throughout the run.All 17 medium/long-cache configurations improved in each of the three ABBA blocks. Aggregated speedups are 1.56–2.56x, corresponding to 36.07%–60.98% lower operator latency. The two 7-token cases are slightly slower: B=1: 0.994x (-0.56% time reduction); B=2: 0.993x (-0.70% time reduction). All 19 timed configurations pass the reference check and produce bit-identical BF16 outputs across both libraries and all 12 processes. These are operator-level measurements; no new end-to-end model speedup is claimed in this run.
Short-cache sample check: one of the six original candidate B=2 processes had a 79.99 us median; the other five were 30.09–30.67 us. That sample remains in the original dataset and median calculation above. A separate ABBA follow-up using the same libraries, GPU7, input seeds and timing settings measured B=1: 29.95 → 30.07 us (0.42% slower); B=2: 30.24 → 30.38 us (0.45% slower). Outputs remained bit-identical. The 79.99 us excursion was not reproduced in that follow-up; no short-cache performance benefit is claimed.
Notes for Reviewers
The optimized path keeps D=128 and aligned BF16 vector loads, accepts page sizes divisible by 256, and uses the existing 4/8 split rule. Type, index contiguity, dimensions, strides, alignment, workspace and ALiBi eligibility are checked before dispatch. Unsupported cases retain the original FlashAttnWithKvcache fallback. Zero-length shards write neutral partials and zero-length sequences return zero.
The Python test helper passes explicit tensor metadata for
cache_seqlens: the generated overload resolver otherwise converts a singleton PyTorch integer tensor to the scalar overload, which this backend does not support. This selects the existing tensor interface without changing bindings or reading lengths back to the host. Python convenience calls have no workspace and retain the existing fallback; the C++ tests and both benchmark roles supply workspace explicitly.Performance analysis is in this PR description; the removed markdown file is no longer referenced by repository code. Host eligibility tests are in
tests/host/iluvatar_flash_decode_policy_test.cpp. Ruff 0.15.22, ClangFormat 21 and whitespace checks passed locally. Maintainer approval is required for upstream workflows on this external contribution.Latest validation
Candidate
d30e6a5incorporates officialmasterf52af91without conflicts. Both clean builds, the candidate host policy test, all 46 candidate GPU tests, the 19-configuration performance comparison, full-repository Ruff 0.15.22 and Clang Format 21 checks, andgit diff --checkpassed. The table above replaces the earlier65579ec/1a68b26measurements with this matched-current-main run. The baseline's two zero-length failures are reported above.Fork CI for
d30e6a5passed: Ruff, Clang Format. Upstream workflows remain pending maintainer approval.