Repository navigation
perf(iluvatar): generalize small-batch BF16 MLP GEMV - #1024
superme1one wants to merge 5 commits into
Conversation
| @@ -0,0 +1,104 @@ | |||
| # Iluvatar small-batch MLP GEMV | |||
There was a problem hiding this comment.
performance analysis doc should not be included in repo.
There was a problem hiding this comment.
Removed docs/performance/iluvatar-mlp-gemv.md from the PR diff in de7b6cb. The current measurements, numerical contract and reproduction details are now in the PR description.
| template <> | ||
| class Operator<Gemm, Device::Type::kIluvatar> | ||
| : public BlasGemm<Blas<Device::Type::kIluvatar>> { | ||
| using Parent = BlasGemm<Blas<Device::Type::kIluvatar>>; |
There was a problem hiding this comment.
Could we rename to Base here?
There was a problem hiding this comment.
Renamed Parent to Base, including the inherited constructors, overload import and fallback call, in de7b6cb.
| if (a.size(1) == 4096) { | ||
| iluvatar::LaunchSmallBatchGemv<4096, 32768>( | ||
| a.data(), b.data(), y.data(), a.size(0), stream); | ||
| } else { | ||
| iluvatar::LaunchSmallBatchGemv<16384, 4096>( | ||
| a.data(), b.data(), y.data(), a.size(0), stream); | ||
| } |
There was a problem hiding this comment.
Here we only support two case, could we add more case to benefit this fast path?
There was a problem hiding this comment.
Expanded this in de7b6cb: K/N are now runtime dimensions rather than two fixed pairs, with separate full-tile and K-tail variants. The gate keeps B1/B2, K divisible by 64, N divisible by 4, the existing dtype/layout/scalar/alignment conditions, and an 8M weight-element threshold to keep small matrices on BLAS. On TG-V200/CoreX, 45 device tests passed with no skips, covering 18 K/N pairs at B1/B2 and fallback cases. The common MLP configurations measured 1.33–2.29x over the original BLAS path in balanced process orders. The original two shapes were also checked against the previous specialization; all four outputs were bit-identical. The full timing table and validation scope are in the updated PR description.
Ruff and Clang Format have both passed for this revision. The upstream workflows are waiting for approval. Could you take another look at the updated implementation and approve the pending workflow runs?
|
|
||
| namespace infini::ops::iluvatar { | ||
|
|
||
| inline bool CanUseSmallBatchGemv(bool bf16, bool same_device, bool has_bias, |
There was a problem hiding this comment.
bf16 is ambiguous, could we rename to all_bf16 or others here?
There was a problem hiding this comment.
Renamed the predicate parameter to all_bf16 in de7b6cb. It explicitly represents the conjunction of the input, weight and output BF16 checks; the condition is unchanged.
Summary
Generalize the Iluvatar BF16 small-batch MLP GEMV implementation from two hard-coded K/N pairs to runtime dimensions. B1/B2 share the same weight-loading and four-accumulator FP32 reduction scheme; full 256-element K tiles and 64-element K tails use separate kernel variants.
The fast path requires B1/B2, K and N at least 1024, K divisible by 64, N divisible by 4, and at least 8,388,608 weight elements. BF16 type, same-device, row-major A/Y, column-major B, aligned pair loads, alpha=1, beta=0 and no-C/no-transpose checks remain in place. Other calls use the original BLAS implementation.
Addresses the review by using
Baseandall_bf16, removing the performance analysis markdown from the repository diff, and expanding shape and fallback coverage.Motivation
The original fixed shapes only accelerated the measured Gate/Up and Down projections. Runtime dimensions cover additional hidden/intermediate sizes and K tails without a model-specific allowlist. Small matrices retain BLAS after measurements showed no useful gain there.
Type of Change
perf— Iluvatar small-batch GEMV optimization.Platforms Affected
WITH_ILUVATAR). Other backend implementations are unchanged.Smoke Test Result
Validated on 2026-11-09 with TG-V200 / CoreX 4.3.0 and native implementation index 0.
Clang Format 21 and
git diff --checkalso passed. Device tests cover 18 K/N shapes at B1/B2, exact bounded-dyadic results, random inputs against FP32 reference (rtol=atol=0.01), alpha fallback, batch/shape boundaries, row-major weights, and padded A/B strides. Host tests cover eligibility and address-alignment rejection.The affected CoreX translation units were rebuilt and the native library relinked in an isolated copy of the previously validated SDK (ABI0). Baseline and candidate use the same Python binding and existing SDK compatibility setup; the candidate changes only the Iluvatar GEMM implementation. Loaded-library paths and source hashes were checked. This is an incremental affected-platform build, not a new all-backend build.
Benchmark / Performance Impact
Standalone InfiniOps GEMM comparison against the unmodified BLAS path at base
1a68b261d19f0ed6f76bbabeab7a25f479f05e07. Same TG-V200, BF16 tensors and native implementation 0; tuning disabled. For each configuration: 20 warmup calls, 5 groups of 100 calls timed with CUDA events, preallocated inputs/output. Baseline→candidate and candidate→baseline process orders were both measured. Table entries average the two per-order medians; allocation, reference calculation and transfers are outside timing. The API submission overhead remains in the event interval.The 14 common MLP K/N pairs (28 batch configurations) improve by 1.33–2.29x. Four additional pairs exercise the 8M-element threshold and all three K-tail lengths. These are operator timings; they are not end-to-end model gains.
Eight below-threshold configurations also remain correct and use BLAS. Their small absolute timings include the extra dispatch-check overhead; no speedup is claimed for them.
Original Specialization Regression
The two original K/N shapes at both batch sizes were separately compared against the pre-review specialization using fixed random inputs and both process orders. All four outputs were bit-identical.
Notes for Reviewers