Skip to content

perf(iluvatar): batch FlashAttnWithKvcache QK reductions - #1026

Open
superme1one wants to merge 7 commits into
InfiniTensor:masterfrom
superme1one:perf/iluvatar-decode-qk-shared-reduction
Open

superme1one wants to merge 7 commits into
InfiniTensor:masterfrom
superme1one:perf/iluvatar-decode-qk-shared-reduction

Conversation

@superme1one

@superme1one superme1one commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

  • Move the Iluvatar batched QK reduction into src/native/cuda/iluvatar/ops/flash_attn_with_kvcache/ and call it through flash_attn_with_kvcache. The PR no longer changes the deprecated PagedAttentionInfinilm implementation.
  • Reduce eight independent QK reductions from 48 barriers to 6, retaining their FP32 addition trees. Use an Iluvatar split/combine implementation with uniform block synchronization and a shared K/V tile.
  • Replace the eligibility function's argument list with DecodeQkConfig; pass kernel inputs in DecodeQkParams.
  • Extend the optimized BF16, head-dimension-128 path to batches 1–16, grouped/multi-head attention, multiple pages and cache layouts. Preserve the existing provider for unsupported configurations.
  • Put native regression coverage in tests/test_flash_attn_with_kvcache.py and remove the standalone performance report from the repository.

Motivation

The original PR optimized only B=1/2, Hq=32, Hkv=2 and a single 4096-token cache page through a deprecated operator. This revision uses the maintained interface and measures additional configurations on TG-V200. The per-operator results below compare this revision with the official baseline; earlier combined-model results remain in companion InfiniLM #610.

Type of Change

  • perf — performance improvement
  • refactor — migrate the implementation to the maintained operator

Platforms Affected

  • Iluvatar (WITH_ILUVATAR)

Smoke Test Result

Validated on 2026-11-16 on TG-V200 GPU7. Both official master f52af91 and candidate d30e6a5 were built from clean source archives in separate directories, using the same CoreX installation and InfiniRT SDK. Both builds use Release, WITH_CPU=ON, WITH_ILUVATAR=ON, WITH_TORCH=OFF, GENERATE_PYTHON_BINDINGS=ON, and INFINI_OPS_OPS=flash_attn_with_kvcache. The candidate host eligibility-policy test passed.

The same candidate test module was run against each native library. The C++ workspace tests use the compiler/prefix helpers in tests/test_cpp_api.py; INFINI_OPS_INSTALL_PREFIX selects the respective isolated installation, with the same INFINI_RT_ROOT and CoreX include directory in CXXFLAGS.

CUDA_VISIBLE_DEVICES=7 IX_VISIBLE_DEVICES=7 INFINI_OPS_ENABLE_AUTO_TUNING=0 \
python -m pytest tests/test_flash_attn_with_kvcache.py --devices iluvatar -q
Candidate: 46 passed, 0 failed, 30 skipped
Official:  44 passed, 2 failed, 30 skipped

The candidate passes all 21 explicit-workspace cases, 24 no-workspace/fallback cases, and the existing paged regression. Coverage includes B=1/2/4/8/16, Hq=8/16/32/64/128, Hkv=1/2/4/8/16, page sizes 256–4096, 4/8 splits, reordered physical pages, per-sequence lengths, padded query/output rows, zero-length sequences, and FP16/D64/B17 fallbacks. The independent reference uses PyTorch FP32 attention with BF16/FP16 output (rtol=2e-2, atol=2e-3).

The official baseline fails only the two added workspace tests containing zero-length sequences: lengths (1, 0) and (0,) (case19 and case20). It returns incorrect values for the zero-length rows. Both pass with the candidate's neutral-partial and zero-output handling. These two boundary cases are excluded from the performance table; all 19 timed configurations pass the reference check for both libraries.

The 30 skips are existing backend/parameter exclusions: Ascend (6), Moore (9), NVIDIA MLA (4), non-paged Iluvatar cases (10), and scalar cache lengths unsupported by the Iluvatar native provider (1).

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
Iluvatar Yes CoreX build and affected native tests passed 46 passed / 30 skipped in the complete affected module; host policy checks passed; full repository suite not run
NVIDIA / MetaX / Cambricon / Moore / Ascend / CPU No N/A — not affected No shared backend implementation changes

Benchmark / Performance Impact

Remeasured on 2026-11-16, TG-V200 GPU7, after merging current official master:

  • Official baseline: f52af91a83f19610b16120ab1be4f5c884c02c1c.
  • Candidate: d30e6a5c3fbd0090d4bb9886162cfaf753f0b279; the merge leaves the Iluvatar fast-path implementation unchanged.
  • BF16, head dimension 128, native implementation 0, automatic tuning disabled, OMP_NUM_THREADS=1. Same device, CoreX toolchain, Release build options, InfiniRT SDK, input seeds, tensor shapes and timing method for both libraries.
  • CoreX driver 4.3.0 and PyTorch 2.4.1. Before every benchmark process, GPU7 reported 0% utilization, a 1500 MHz SM clock, and a temperature of 33–34 C.
  • Isolated native library SHA256: official 4d5c5356cff8a8f652fa1f9c0eb192b1b61be988792d8482f3018c6bad742553; candidate bf3bd163fddb6697438b4dafcf53165bf78a810eb529bfcab9ebf01eb400f665. Loaded library paths were checked in every benchmark process.

Inputs, output and the 8-split workspace are preallocated. Both roles invoke the public C++ FlashAttnWithKvcache::Call with the same explicit workspace and stream. Each process warms up 20 calls and measures five sets of 100 calls using CUDA events. Three serial official → candidate → candidate → official blocks give six processes and 30 timing samples per role/configuration. Each table entry is the median of six per-process medians. Cache lengths decrease by the batch-row index; the table shows the first row's length. GPU status was recorded throughout the run.

All 17 medium/long-cache configurations improved in each of the three ABBA blocks. Aggregated speedups are 1.56–2.56x, corresponding to 36.07%–60.98% lower operator latency. The two 7-token cases are slightly slower: B=1: 0.994x (-0.56% time reduction); B=2: 0.993x (-0.70% time reduction). All 19 timed configurations pass the reference check and produce bit-identical BF16 outputs across both libraries and all 12 processes. These are operator-level measurements; no new end-to-end model speedup is claimed in this run.

B / Q heads / KV heads Cache length / page size / pages Layout Official, us Candidate, us Speedup Time reduction
1 / 32 / 2 7 / 4096 / 1 head_first 29.99 30.16 0.994x -0.56%
2 / 32 / 2 7 / 4096 / 1 head_first 30.29 30.50 0.993x -0.70%
1 / 32 / 2 2049 / 4096 / 1 head_first 582.73 329.18 1.770x 43.51%
2 / 32 / 2 2049 / 4096 / 1 head_first 584.45 332.45 1.758x 43.12%
1 / 32 / 2 2910 / 4096 / 1 head_first 806.52 449.53 1.794x 44.26%
2 / 32 / 2 2910 / 4096 / 1 head_first 808.41 453.68 1.782x 43.88%
1 / 32 / 2 4096 / 4096 / 1 head_first 1124.13 622.66 1.805x 44.61%
2 / 32 / 2 4096 / 4096 / 1 head_first 1126.43 627.00 1.797x 44.34%
1 / 8 / 1 777 / 256 / 4 token_first 241.43 141.57 1.705x 41.36%
2 / 16 / 4 1537 / 512 / 4 token_first 445.61 255.48 1.744x 42.67%
4 / 32 / 8 4096 / 256 / 16 token_first 1270.88 795.34 1.598x 37.42%
8 / 16 / 2 2910 / 512 / 8 head_first 848.49 520.46 1.630x 38.66%
16 / 8 / 1 1025 / 256 / 8 token_first 320.59 204.95 1.564x 36.07%
1 / 64 / 8 4095 / 1024 / 4 head_first 1126.15 627.40 1.795x 44.29%
2 / 128 / 16 2049 / 2048 / 2 token_first 1179.56 460.24 2.563x 60.98%
4 / 8 / 8 513 / 256 / 4 token_first 169.86 105.67 1.607x 37.79%
1 / 32 / 2 8191 / 256 / 32 token_first 1131.54 627.28 1.804x 44.56%
4 / 16 / 4 8192 / 256 / 32 head_first 1269.94 790.31 1.607x 37.77%
4 / 8 / 2 2047 / 512 / 4 padded_rows 581.09 329.15 1.765x 43.36%

Short-cache sample check: one of the six original candidate B=2 processes had a 79.99 us median; the other five were 30.09–30.67 us. That sample remains in the original dataset and median calculation above. A separate ABBA follow-up using the same libraries, GPU7, input seeds and timing settings measured B=1: 29.95 → 30.07 us (0.42% slower); B=2: 30.24 → 30.38 us (0.45% slower). Outputs remained bit-identical. The 79.99 us excursion was not reproduced in that follow-up; no short-cache performance benefit is claimed.

Notes for Reviewers

The optimized path keeps D=128 and aligned BF16 vector loads, accepts page sizes divisible by 256, and uses the existing 4/8 split rule. Type, index contiguity, dimensions, strides, alignment, workspace and ALiBi eligibility are checked before dispatch. Unsupported cases retain the original FlashAttnWithKvcache fallback. Zero-length shards write neutral partials and zero-length sequences return zero.

The Python test helper passes explicit tensor metadata for cache_seqlens: the generated overload resolver otherwise converts a singleton PyTorch integer tensor to the scalar overload, which this backend does not support. This selects the existing tensor interface without changing bindings or reading lengths back to the host. Python convenience calls have no workspace and retain the existing fallback; the C++ tests and both benchmark roles supply workspace explicitly.

Performance analysis is in this PR description; the removed markdown file is no longer referenced by repository code. Host eligibility tests are in tests/host/iluvatar_flash_decode_policy_test.cpp. Ruff 0.15.22, ClangFormat 21 and whitespace checks passed locally. Maintainer approval is required for upstream workflows on this external contribution.

Latest validation

Candidate d30e6a5 incorporates official master f52af91 without conflicts. Both clean builds, the candidate host policy test, all 46 candidate GPU tests, the 19-configuration performance comparison, full-repository Ruff 0.15.22 and Clang Format 21 checks, and git diff --check passed. The table above replaces the earlier 65579ec / 1a68b26 measurements with this matched-current-main run. The baseline's two zero-length failures are reported above.

Fork CI for d30e6a5 passed: Ruff, Clang Format. Upstream workflows remain pending maintainer approval.

@superme1one
superme1one marked this pull request as ready for review November 6, 2026 03:16
@superme1one
superme1one requested a review from a team November 6, 2026 03:16
Comment thread tests/test_iluvatar_decode_reduction.py Outdated
[length - i for i in range(batch)], dtype=torch.int32, device=device
)
out = torch.empty_like(q)
infini.ops.paged_attention_infinilm(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This op is deprecated, plz don't use it anymore, you could use flash_attn_with_kvcache instead.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the tests to call infini.ops.flash_attn_with_kvcache with native implementation 0. The deprecated operator is no longer used by this PR’s tests. All 46 affected native cases passed on TG-V200 without skips.

Comment thread tests/test_iluvatar_decode_reduction.py Outdated
@@ -0,0 +1,56 @@
import infini.ops

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz move this test to test_{ops}.py file.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the regressions into the existing tests/test_flash_attn_with_kvcache.py and removed tests/test_iluvatar_decode_reduction.py. It now contains 24 smoke-marked Python no-workspace/fallback cases and 21 C++ workspace cases for the optimized path. Together with the existing paged regression, all 46 passed on TG-V200. The C++ tests use the existing SDK/compiler test helpers; the command and environment requirements are in the PR description.

@@ -0,0 +1,91 @@
# Iluvatar Decode QK reduction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc is not necessary.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed docs/performance/iluvatar-decode-qk-reduction.md from the PR. The updated measurement method and per-configuration results are now in the PR description.

Comment on lines +6 to +11
bool iluvatar, bool bf16, bool int32, std::size_t batch, std::size_t heads,
std::size_t kv, std::size_t dim, std::size_t blocks, std::size_t block,
int splits, bool alibi, std::ptrdiff_t qs, std::ptrdiff_t qh,
std::ptrdiff_t os, std::ptrdiff_t oh, std::ptrdiff_t kb, std::ptrdiff_t ks,
std::ptrdiff_t kh, std::ptrdiff_t vb, std::ptrdiff_t vs,
std::ptrdiff_t vh) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This input list is too long, could we use struct to reduce input number here?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the positional eligibility arguments with DecodeQkConfig, which groups type, shape, stride and alignment metadata. Kernel inputs are passed through DecodeQkParams. Added host coverage for the eligibility boundaries and split selection.

k_cache_slot_stride_, k_cache_head_stride_,
v_cache_block_stride_, v_cache_slot_stride_,
v_cache_head_stride_, num_splits);
const auto launch_split = [&](auto batched) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This op is deprecated link, we don't expect any change on it.
If you need this kernel, please migrate this kernel to this folder

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Migrated the optimized split/combine kernels to src/native/cuda/iluvatar/ops/flash_attn_with_kvcache/ and wired them into Operator<FlashAttnWithKvcache, kIluvatar>. All changes to paged_attention_infinilm have been removed from the PR diff. The new path builds with CoreX and passed the 46 native tests through the maintained interface; its existing fallback is unchanged.

Comment on lines +12 to +16
return iluvatar && bf16 && int32 && (batch == 1 || batch == 2) &&
heads == 32 && kv == 2 && dim == 128 && blocks == 1 && block == 4096 &&
splits == 4 && !alibi && qs == 4096 && qh == 128 && os == 4096 &&
oh == 128 && kb == 1048576 && ks == 128 && kh == 524288 &&
vb == 1048576 && vs == 128 && vh == 524288;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that our change take effect for one specifice case, could it improve performance for other case?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expanded beyond the original single layout: B=1/2/4/8/16, Hq=8/16/32/64/128, Hkv=1/2/4/8/16, page sizes 256–4096, multiple pages, 4/8 splits, MHA/GQA and padded query/output rows. With explicit workspace through the public C++ interface, the 17 medium/long-cache configurations improved in both same-card role orders by 1.57–2.58x. The two 7-token cases were within 1% of baseline in the averaged measurements. The PR description includes every configuration and latency. All 19 configurations produced bit-identical BF16 outputs across the official and candidate libraries in both role orders. D=128 remains the optimized head dimension, with D64/FP16/B17 fallback coverage. Please take another look; the upstream workflows still need maintainer approval.

@superme1one superme1one changed the title perf(iluvatar): batch Decode Attention QK reductions perf(iluvatar): batch FlashAttnWithKvcache QK reductions Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants