Skip to content

perf(iluvatar): fuse causal Softmax with FP32 accumulation - #1025

Open
superme1one wants to merge 5 commits into
InfiniTensor:masterfrom
superme1one:perf/iluvatar-causal-softmax-fp32
Open

superme1one wants to merge 5 commits into
InfiniTensor:masterfrom
superme1one:perf/iluvatar-causal-softmax-fp32

Conversation

@superme1one

@superme1one superme1one commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

Fuse causal masking and Softmax for eligible contiguous 3D Iluvatar BF16 inputs with 2 <= Q <= K <= 4096. Reduce max/exp/sum in FP32, then divide the BF16-rounded exponent by the unrounded FP32 sum to preserve the existing InfiniLM numerical contract.

Review revision validation (2026-11-09)

  • Removed docs/performance/iluvatar-causal-softmax-fp32.md from the repository diff; numerical behavior, reproduction commands and results are recorded in this description.
  • Moved all eight Iluvatar native regression cases into tests/test_causal_softmax.py, preserving the four Q/K pairs, in-place/out-of-place modes, masked NaNs and exact BF16 reference comparison. The moved cases explicitly select implementation 0 and are included in the smoke selection.
  • Renamed the input/output BF16 predicate to all_bf16 and the base-class alias to Base, including inherited constructors and the fallback call. The kernel and dispatch conditions are unchanged by this revision.

Rebuilt the three affected CoreX translation units and relinked the native library using the previously validated SDK, ABI0 and the existing compatibility setup. On TG-V200/CoreX 4.3.0, the host policy test passed and both affected test modules passed: 44 passed, 0 failed, 0 skipped. All eight moved cases were also confirmed to carry the smoke marker. Ruff 0.15.22 (175 Python files), Clang Format 21 and git diff --check passed locally.

c++ -std=c++17 -Isrc tests/host/iluvatar_causal_policy_test.cpp -o /tmp/causal-policy
/tmp/causal-policy
python -m pytest tests/test_causal_softmax.py tests/test_causal_softmax_infinilm.py --devices iluvatar -q

The targeted native regression compares fused causal masking plus Softmax against a materialized causal mask and the backend Softmax. It checks exact BF16 output, finite values after masked-NaN selection, and zero masked positions, for both output modes. Other shapes/layouts/types retain the existing provider. The dated model measurements below are retained from the prior validation.

Prior complete CoreX build and GPU verification (2026-11-06)

CoreX current-code validation completed on 2026-11-06 against InfiniLM
12a4492f70dd33235e3d142e333ad4a9629d3b7b and InfiniOps
1a68b261d19f0ed6f76bbabeab7a25f479f05e07. Both complete SDKs compiled
and imported. Both roles used the identical documented CoreX validation adapter
and ABI0, with automatic tuning disabled. The adapter includes existing CUDA
Argmax/Triu/Tril registrations, unchanged rejecting unavailable factories, a
copy-on-write read-only-view loader, and the existing paged-only FlashVarlen
provider's metadata gate. See the companion InfiniLM
docs/performance/corex-validation/README.md for exact scope.

The affected native GPU tests passed without skips: GEMV 4, causal Softmax 8,
Decode Attention 8, plus 7 common Argmax/mask checks. The submitted device tests
now explicitly bind implementation index 0. GEMV fallback uses the supported
no-C, alpha=0.5 contract; this backend rejects C/bias at its base operator.

Original tasks 554/172 ran on one TG-V200 in forward and reverse role order.
Each of the four model processes used one simultaneous two-request warmup wave
and one simultaneous two-request measured wave: 16 requests total, 8 measured.
All 16 answers were correct with natural EOS, zero errors/truncations, normal
scope return, final library maps, atexit and actual OS exit0. The original
512-token greedy budget, EOS=[2,73440], strict last-#### numeric scorer, inputs
and weights were unchanged. Two physical rows, full Prefill and graph off were
held fixed. No new independent quality questions were added.

Mean two-request measured-wave time was 14.976775177 ->
9.782192262 seconds (-34.6843%)
.
Completed request throughput was 0.133540096 -> 0.204453148
requests/s (+53.1024%)
. Published token throughput
was 20.631944884 -> 31.588011330 tokens/s and is reported separately.
The time denominator includes queue and text processing and excludes loading,
warmup and final shutdown. Both role orders reduced application time; this is
the combined four-optimization gain, not an individual-PR or component sum.

Order Official wave, s Combined wave, s Time reduction
forward 14.973374084 9.782256734 34.6690%
reverse 14.980176271 9.782127790 34.6995%

Token-different task IDs across the two measured orders: none. Answers and token differences are recorded separately.
Raw clocks, output IDs, strict scores, native receipts and ordinary exits were independently recomputed. This two-question integration check is kept separate from the historical 100-question experiment below.

The fused kernel uses the existing InfiniLM order: FP32 max/exp/sum, BF16 exponent storage, then division by the unrounded FP32 sum. It creates no materialized mask tensor or host synchronization. Machine-readable combined evidence and the common validation adapter are in companion InfiniLM #610; native fixture commits are recorded in that evidence.

Historical 100-question quality and performance evidence

Historical four-optimization TG-V200/CoreX result: 100 fixed new GSM8K targets, max_tokens=512, greedy EOS=[2,73440], unchanged strict last-#### numeric scoring. Official accuracy 82/100 → 84/100; all requests completed/parsed, zero errors and truncations. 34/100 token-different, including one correct→wrong (1097: 480→448, reference480) and three wrong→correct (672,1259,1203).

Sum of per-version same-card formal paired-wave makespans 749.203818 → 505.326616 s (-32.5515%); completed request throughput +48.2613%. Published output tokens 13894→14213, so token throughput +51.6653% is reported separately. Each target's two versions used the same physical card; first20 serial, remaining80 sharded over two cards. Load/warmup/final shutdown excluded; queue/text processing included. These are combined gains, not individual-PR gains or the parallel run's wall time. Eight demonstrations came from a disjoint test-corpus subset (internal paired evaluation).

Historical official InfiniLM baseline: 6e7f9537173cd33bb30539ebf8d92ab6d869660d. The current-code two-task result above is a separate experiment.

Notes for Reviewers

Unsupported metadata retains the original provider fallback. This provider is the companion dependency of InfiniLM #610. Native tests explicitly bind implementation index 0.

The four PRs form the measured combination. This validation does not attribute the total gain to any one PR. The original historical individual regression remains disclosed above.

Platforms and checks

Check Result
CoreX complete official/combined SDK build and import Passed
Affected Iluvatar native GPU regressions 20/20 passed, no skips
Common Argmax/triangular-mask GPU checks 7/7 passed, no skips
Same-card original two-task application, forward/reverse order Passed, all 16 requests correct
Ordinary scope/maps/atexit/OS exit Passed
Other hardware backends Not executed in this validation

Related optimization PRs

Checklist

  • Focused implementation and conservative fallback.
  • Current CoreX build and native GPU checks.
  • Original two-task same-card quality and application performance check.
  • Original scoring, inputs and generation budget retained.
  • Runtime exits and raw measurements independently verified.
  • Documentation preserves current and historical evidence separately.
  • Diff whitespace check passed; no credentials, weights or private server addresses in the PR.

@superme1one
superme1one marked this pull request as ready for review November 6, 2026 03:15
@superme1one
superme1one requested a review from a team November 6, 2026 03:15
@@ -0,0 +1,85 @@
# Iluvatar Prefill causal Softmax

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance doc should not be included in repo.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed docs/performance/iluvatar-causal-softmax-fp32.md from the PR diff in 2ac1c49. The numerical contract, reproduction commands and dated validation results are now in the PR description.

Comment thread tests/test_iluvatar_causal_fp32.py Outdated
@@ -0,0 +1,39 @@
import infini.ops

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plz move this test to test file

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved all eight native Iluvatar cases into tests/test_causal_softmax.py and removed the separate test file in 2ac1c49. The original Q/K pairs, both output modes, masked-NaN checks, exact BF16 comparison and explicit implementation index 0 are preserved; the cases are now marked for smoke runs. After rebuilding the affected CoreX objects, both causal Softmax test modules passed on TG-V200: 44 passed, 0 failed, 0 skipped. All eight moved cases were confirmed in the smoke selection.


namespace infini::ops::iluvatar {

inline bool CanUseCausalSoftmaxFp32(bool bf16, bool same_device,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment, plz refer to pr #1024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed bf16 to all_bf16 in 2ac1c49 to make the input/output dtype requirement explicit. Also applied the Base alias convention from #1024 to this provider, including the inherited constructors and fallback call. Dispatch conditions and kernel calculations are unchanged. The host policy test and 44 device regressions passed. Could you take another look at this revision and approve the pending upstream workflow runs?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants