Repository navigation
perf(iluvatar): fuse causal Softmax with FP32 accumulation - #1025
superme1one wants to merge 5 commits into
Conversation
| @@ -0,0 +1,85 @@ | |||
| # Iluvatar Prefill causal Softmax | |||
There was a problem hiding this comment.
performance doc should not be included in repo.
There was a problem hiding this comment.
Removed docs/performance/iluvatar-causal-softmax-fp32.md from the PR diff in 2ac1c49. The numerical contract, reproduction commands and dated validation results are now in the PR description.
| @@ -0,0 +1,39 @@ | |||
| import infini.ops | |||
There was a problem hiding this comment.
Moved all eight native Iluvatar cases into tests/test_causal_softmax.py and removed the separate test file in 2ac1c49. The original Q/K pairs, both output modes, masked-NaN checks, exact BF16 comparison and explicit implementation index 0 are preserved; the cases are now marked for smoke runs. After rebuilding the affected CoreX objects, both causal Softmax test modules passed on TG-V200: 44 passed, 0 failed, 0 skipped. All eight moved cases were confirmed in the smoke selection.
|
|
||
| namespace infini::ops::iluvatar { | ||
|
|
||
| inline bool CanUseCausalSoftmaxFp32(bool bf16, bool same_device, |
There was a problem hiding this comment.
Renamed bf16 to all_bf16 in 2ac1c49 to make the input/output dtype requirement explicit. Also applied the Base alias convention from #1024 to this provider, including the inherited constructors and fallback call. Dispatch conditions and kernel calculations are unchanged. The host policy test and 44 device regressions passed. Could you take another look at this revision and approve the pending upstream workflow runs?
Summary
Fuse causal masking and Softmax for eligible contiguous 3D Iluvatar BF16 inputs with 2 <= Q <= K <= 4096. Reduce max/exp/sum in FP32, then divide the BF16-rounded exponent by the unrounded FP32 sum to preserve the existing InfiniLM numerical contract.
Review revision validation (2026-11-09)
docs/performance/iluvatar-causal-softmax-fp32.mdfrom the repository diff; numerical behavior, reproduction commands and results are recorded in this description.tests/test_causal_softmax.py, preserving the four Q/K pairs, in-place/out-of-place modes, masked NaNs and exact BF16 reference comparison. The moved cases explicitly select implementation 0 and are included in the smoke selection.all_bf16and the base-class alias toBase, including inherited constructors and the fallback call. The kernel and dispatch conditions are unchanged by this revision.Rebuilt the three affected CoreX translation units and relinked the native library using the previously validated SDK, ABI0 and the existing compatibility setup. On TG-V200/CoreX 4.3.0, the host policy test passed and both affected test modules passed: 44 passed, 0 failed, 0 skipped. All eight moved cases were also confirmed to carry the smoke marker. Ruff 0.15.22 (175 Python files), Clang Format 21 and
git diff --checkpassed locally.The targeted native regression compares fused causal masking plus Softmax against a materialized causal mask and the backend Softmax. It checks exact BF16 output, finite values after masked-NaN selection, and zero masked positions, for both output modes. Other shapes/layouts/types retain the existing provider. The dated model measurements below are retained from the prior validation.
Prior complete CoreX build and GPU verification (2026-11-06)
CoreX current-code validation completed on 2026-11-06 against InfiniLM
12a4492f70dd33235e3d142e333ad4a9629d3b7band InfiniOps1a68b261d19f0ed6f76bbabeab7a25f479f05e07. Both complete SDKs compiledand imported. Both roles used the identical documented CoreX validation adapter
and ABI0, with automatic tuning disabled. The adapter includes existing CUDA
Argmax/Triu/Tril registrations, unchanged rejecting unavailable factories, a
copy-on-write read-only-view loader, and the existing paged-only FlashVarlen
provider's metadata gate. See the companion InfiniLM
docs/performance/corex-validation/README.mdfor exact scope.The affected native GPU tests passed without skips: GEMV 4, causal Softmax 8,
Decode Attention 8, plus 7 common Argmax/mask checks. The submitted device tests
now explicitly bind implementation index 0. GEMV fallback uses the supported
no-C, alpha=0.5 contract; this backend rejects C/bias at its base operator.
Original tasks 554/172 ran on one TG-V200 in forward and reverse role order.
Each of the four model processes used one simultaneous two-request warmup wave
and one simultaneous two-request measured wave: 16 requests total, 8 measured.
All 16 answers were correct with natural EOS, zero errors/truncations, normal
scope return, final library maps, atexit and actual OS exit0. The original
512-token greedy budget, EOS=[2,73440], strict last-#### numeric scorer, inputs
and weights were unchanged. Two physical rows, full Prefill and graph off were
held fixed. No new independent quality questions were added.
Mean two-request measured-wave time was 14.976775177 ->
9.782192262 seconds (-34.6843%).
Completed request throughput was 0.133540096 -> 0.204453148
requests/s (+53.1024%). Published token throughput
was 20.631944884 -> 31.588011330 tokens/s and is reported separately.
The time denominator includes queue and text processing and excludes loading,
warmup and final shutdown. Both role orders reduced application time; this is
the combined four-optimization gain, not an individual-PR or component sum.
Token-different task IDs across the two measured orders: none. Answers and token differences are recorded separately.
Raw clocks, output IDs, strict scores, native receipts and ordinary exits were independently recomputed. This two-question integration check is kept separate from the historical 100-question experiment below.
The fused kernel uses the existing InfiniLM order: FP32 max/exp/sum, BF16 exponent storage, then division by the unrounded FP32 sum. It creates no materialized mask tensor or host synchronization. Machine-readable combined evidence and the common validation adapter are in companion InfiniLM #610; native fixture commits are recorded in that evidence.
Historical 100-question quality and performance evidence
Historical four-optimization TG-V200/CoreX result: 100 fixed new GSM8K targets, max_tokens=512, greedy EOS=[2,73440], unchanged strict last-
####numeric scoring. Official accuracy 82/100 → 84/100; all requests completed/parsed, zero errors and truncations. 34/100 token-different, including one correct→wrong (1097: 480→448, reference480) and three wrong→correct (672,1259,1203).Sum of per-version same-card formal paired-wave makespans 749.203818 → 505.326616 s (-32.5515%); completed request throughput +48.2613%. Published output tokens 13894→14213, so token throughput +51.6653% is reported separately. Each target's two versions used the same physical card; first20 serial, remaining80 sharded over two cards. Load/warmup/final shutdown excluded; queue/text processing included. These are combined gains, not individual-PR gains or the parallel run's wall time. Eight demonstrations came from a disjoint test-corpus subset (internal paired evaluation).
Historical official InfiniLM baseline:
6e7f9537173cd33bb30539ebf8d92ab6d869660d. The current-code two-task result above is a separate experiment.Notes for Reviewers
Unsupported metadata retains the original provider fallback. This provider is the companion dependency of InfiniLM #610. Native tests explicitly bind implementation index 0.
The four PRs form the measured combination. This validation does not attribute the total gain to any one PR. The original historical individual regression remains disclosed above.
Platforms and checks
Related optimization PRs
Checklist