Skip to content

perf(iluvatar): generalize small-batch BF16 MLP GEMV - #1024

Open
superme1one wants to merge 5 commits into
InfiniTensor:masterfrom
superme1one:perf/iluvatar-small-batch-mlp-gemv
Open

superme1one wants to merge 5 commits into
InfiniTensor:masterfrom
superme1one:perf/iluvatar-small-batch-mlp-gemv

Conversation

@superme1one

@superme1one superme1one commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

Generalize the Iluvatar BF16 small-batch MLP GEMV implementation from two hard-coded K/N pairs to runtime dimensions. B1/B2 share the same weight-loading and four-accumulator FP32 reduction scheme; full 256-element K tiles and 64-element K tails use separate kernel variants.

The fast path requires B1/B2, K and N at least 1024, K divisible by 64, N divisible by 4, and at least 8,388,608 weight elements. BF16 type, same-device, row-major A/Y, column-major B, aligned pair loads, alpha=1, beta=0 and no-C/no-transpose checks remain in place. Other calls use the original BLAS implementation.

Addresses the review by using Base and all_bf16, removing the performance analysis markdown from the repository diff, and expanding shape and fallback coverage.

Motivation

The original fixed shapes only accelerated the measured Gate/Up and Down projections. Runtime dimensions cover additional hidden/intermediate sizes and K tails without a model-specific allowlist. Small matrices retain BLAS after measurements showed no useful gain there.

Type of Change

  • perf — Iluvatar small-batch GEMV optimization.

Platforms Affected

  • Iluvatar (WITH_ILUVATAR). Other backend implementations are unchanged.

Smoke Test Result

Validated on 2026-11-09 with TG-V200 / CoreX 4.3.0 and native implementation index 0.

g++ -std=c++17 -Isrc tests/host/iluvatar_gemv_policy_test.cpp -o gemv-policy
./gemv-policy
# passed
python -m pytest tests/test_iluvatar_mlp_gemv.py --devices iluvatar -q
# 45 passed, 0 failed, 0 skipped
ruff check .
ruff format --check .
# passed, 176 files formatted

Clang Format 21 and git diff --check also passed. Device tests cover 18 K/N shapes at B1/B2, exact bounded-dyadic results, random inputs against FP32 reference (rtol=atol=0.01), alpha fallback, batch/shape boundaries, row-major weights, and padded A/B strides. Host tests cover eligibility and address-alignment rejection.

The affected CoreX translation units were rebuilt and the native library relinked in an isolated copy of the previously validated SDK (ABI0). Baseline and candidate use the same Python binding and existing SDK compatibility setup; the candidate changes only the Iluvatar GEMM implementation. Loaded-library paths and source hashes were checked. This is an incremental affected-platform build, not a new all-backend build.

Platform Affected Build / Smoke Result Full Result / Notes
Iluvatar Yes Incremental build and affected GEMV smoke passed 45 device tests plus host eligibility tests
NVIDIA / MetaX / Cambricon / Moore / Ascend No N/A — unchanged Not run for this revision

Benchmark / Performance Impact

Standalone InfiniOps GEMM comparison against the unmodified BLAS path at base 1a68b261d19f0ed6f76bbabeab7a25f479f05e07. Same TG-V200, BF16 tensors and native implementation 0; tuning disabled. For each configuration: 20 warmup calls, 5 groups of 100 calls timed with CUDA events, preallocated inputs/output. Baseline→candidate and candidate→baseline process orders were both measured. Table entries average the two per-order medians; allocation, reference calculation and transfers are outside timing. The API submission overhead remains in the event interval.

The 14 common MLP K/N pairs (28 batch configurations) improve by 1.33–2.29x. Four additional pairs exercise the 8M-element threshold and all three K-tail lengths. These are operator timings; they are not end-to-end model gains.

K N B1 BLAS → GEMV (µs) B1 speedup B2 BLAS → GEMV (µs) B2 speedup
2048 11008 96.17 → 49.88 1.93x 97.29 → 55.09 1.77x
5504 2048 57.51 → 30.35 1.90x 46.96 → 35.39 1.33x
3072 16384 218.58 → 105.93 2.06x 218.83 → 105.47 2.07x
8192 3072 137.31 → 63.02 2.18x 137.58 → 73.76 1.87x
4096 22016 392.93 → 208.21 1.89x 393.26 → 201.51 1.95x
11008 4096 200.23 → 96.17 2.08x 200.15 → 107.17 1.87x
4096 28672 510.66 → 268.93 1.90x 510.73 → 255.30 2.00x
14336 4096 270.16 → 123.09 2.19x 269.96 → 141.02 1.91x
5120 27648 577.24 → 283.20 2.04x 577.21 → 268.91 2.15x
13824 5120 297.81 → 151.50 1.97x 298.05 → 212.99 1.40x
8192 57344 1966.76 → 915.23 2.15x 1967.09 → 946.00 2.08x
28672 8192 1108.72 → 483.55 2.29x 1111.32 → 535.83 2.07x
4096 32768 569.09 → 303.66 1.87x 569.12 → 291.46 1.95x
16384 4096 281.36 → 139.74 2.01x 281.11 → 155.58 1.81x
2048 4096 70.58 → 70.37 1.00x 51.29 → 51.25 1.00x
4160 2052 42.09 → 41.61 1.01x 37.81 → 35.91 1.05x
4224 2056 37.95 → 33.20 1.14x 37.92 → 29.67 1.28x
4288 2060 38.37 → 27.47 1.40x 38.25 → 29.44 1.30x

Eight below-threshold configurations also remain correct and use BLAS. Their small absolute timings include the extra dispatch-check overhead; no speedup is claimed for them.

Original Specialization Regression

The two original K/N shapes at both batch sizes were separately compared against the pre-review specialization using fixed random inputs and both process orders. All four outputs were bit-identical.

B K N Original (µs) Expanded (µs) Speedup
1 4096 32768 314.47 304.12 1.034x
2 4096 32768 294.23 290.38 1.013x
1 16384 4096 150.73 139.72 1.079x
2 16384 4096 159.30 156.43 1.018x

Notes for Reviewers

  • FP32 reduction order differs from BLAS; the optimization preserves a tolerance-based numerical contract, not general bitwise equivalence to BLAS.
  • Two extra probes with 2-byte-offset BF16 storage produced incorrect results in both the unmodified BLAS baseline and this candidate. The existing 4-byte-alignment gate is retained and checked on the host; correctness of that pre-existing vendor-path case is not claimed by this PR. The passing device fallback tests use supported layouts/strides.
  • The earlier model-level 100-question experiment combined this PR with three other optimizations: 82/100 → 84/100, 34 token-different outputs, including one correct→wrong. That historical combined result is distinct from the standalone operator measurements above; no new model-quality claim is made for the expanded shapes.
  • The performance analysis document has been removed from the code diff. Current measurements and reproduction details are kept here for review.

@superme1one
superme1one marked this pull request as ready for review November 6, 2026 03:15
@superme1one
superme1one requested a review from a team November 6, 2026 03:15
Comment thread docs/performance/iluvatar-mlp-gemv.md Outdated
@@ -0,0 +1,104 @@
# Iluvatar small-batch MLP GEMV

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance analysis doc should not be included in repo.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed docs/performance/iluvatar-mlp-gemv.md from the PR diff in de7b6cb. The current measurements, numerical contract and reproduction details are now in the PR description.

template <>
class Operator<Gemm, Device::Type::kIluvatar>
: public BlasGemm<Blas<Device::Type::kIluvatar>> {
using Parent = BlasGemm<Blas<Device::Type::kIluvatar>>;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we rename to Base here?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed Parent to Base, including the inherited constructors, overload import and fallback call, in de7b6cb.

Comment on lines +40 to +46
if (a.size(1) == 4096) {
iluvatar::LaunchSmallBatchGemv<4096, 32768>(
a.data(), b.data(), y.data(), a.size(0), stream);
} else {
iluvatar::LaunchSmallBatchGemv<16384, 4096>(
a.data(), b.data(), y.data(), a.size(0), stream);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we only support two case, could we add more case to benefit this fast path?

@superme1one superme1one Nov 9, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expanded this in de7b6cb: K/N are now runtime dimensions rather than two fixed pairs, with separate full-tile and K-tail variants. The gate keeps B1/B2, K divisible by 64, N divisible by 4, the existing dtype/layout/scalar/alignment conditions, and an 8M weight-element threshold to keep small matrices on BLAS. On TG-V200/CoreX, 45 device tests passed with no skips, covering 18 K/N pairs at B1/B2 and fallback cases. The common MLP configurations measured 1.33–2.29x over the original BLAS path in balanced process orders. The original two shapes were also checked against the previous specialization; all four outputs were bit-identical. The full timing table and validation scope are in the updated PR description.

Ruff and Clang Format have both passed for this revision. The upstream workflows are waiting for approval. Could you take another look at the updated implementation and approve the pending workflow runs?


namespace infini::ops::iluvatar {

inline bool CanUseSmallBatchGemv(bool bf16, bool same_device, bool has_bias,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bf16 is ambiguous, could we rename to all_bf16 or others here?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the predicate parameter to all_bf16 in de7b6cb. It explicitly represents the conjunction of the input, weight and output BF16 checks; the condition is unchanged.

@superme1one superme1one changed the title perf(iluvatar): specialize BF16 Gate-Up and Down GEMV perf(iluvatar): generalize small-batch BF16 MLP GEMV Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants