Repository navigation
Conversation
Convert eligible weights with aclnnTransMatmulWeight and execute them through aclnnMatmulWeightNz.
Cache the repeatable MatmulWeightNz executor and qualify the hot Ascend decode kernels for device graph capture. This ports the capture behavior used by the FM9G70B Ascend stack to the current operator API.
wooway777
force-pushed
the
merge/fm9g70-ascend-stack
branch
from
November 16, 2026 07:41
374fa17 to
b956fdd
Compare
Port the CANN FAI decode path, parallelize and chunk paged cache writes, and retain host-side uniform prefill sequence metadata to reduce synchronous D2H copies.
wooway777
force-pushed
the
merge/fm9g70-ascend-stack
branch
from
November 16, 2026 08:02
b956fdd to
f7f619c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ports and productizes the FM9G70 Ascend serving stack. It contains three intentionally ordered commits:
The change is Ascend-only. Existing generic and non-Ascend paths remain unchanged, and unsupported Ascend layouts fall back to the previous implementations.
Motivation
FM9G70B BF16 MHA serving on Ascend 910B3 was bottlenecked by four separate costs:
reshape_and_cache_flash_stridedlaunched one serial work block for every token/head and consumed about 36.8% of device time per rank (average 7.55 ms in the profiled serving shape).This PR supplies the missing layout operators and replaces the hot FM9G70 paged decode/cache-write paths with specialized Ascend implementations while retaining guarded fallbacks.
Change Intent by Commit
1.
f91d31f1: "feat(ascend): add FRACTAL_NZ weight layout support"Intent: provide an explicit, shape-checked operator pair for converting and consuming FRACTAL_NZ dense weights.
src/base/matmul_weight_nz.hMatmulWeightNz(input, weight, out).src/base/trans_matmul_weight_nz.hTransMatmulWeightNz(input, output).src/native/ascend/ops/trans_matmul_weight_nz/kernel.haclnnTransMatmulWeightNz.src/native/ascend/ops/matmul_weight_nz/kernel.hMatmulWeightNzwithaclnnMatmulWeightNz.ACL_FORMAT_FRACTAL_NZfor the converted weight while keeping input/output descriptors contiguous.tests/test_matmul_weight_nz.pytorch.matmul.2.
3ad9b36e: "perf(ascend): mark FM9G hot kernels graph-capture safe"Intent: reduce artificial graph segmentation for the FM9G70 capture-enabled serving graph.
The following wrappers now override
capture_safe()totrue:addargmaxcopyembeddingfillflash_attn_varlen_funcflash_attn_with_kvcachefused_add_rms_normgemmlinearmatmulmatmul_weight_nzmulreshape_and_cache_flashrms_normrotary_embeddingsilu_and_mulThese operators launch through ACLNN or the existing generated launchers on the caller's stream and do not require host-side work between capture and replay. Marking them capture-safe allows larger device graph segments and reduces per-layer graph boundaries.
src/native/ascend/ops/embedding/kernel.halso contains formatting-only adjustment around a deprecated constructor; there is no behavior change beyondcapture_safe().src/native/ascend/ops/matmul_weight_nz/kernel.his changed more substantially:aclOpExecutorare retained after the first invocation.aclSetAclOpExecutorRepeatablemarks the executor reusable.aclSetRawTensorAddrto bind new input/weight/output addresses instead of recreating descriptors.3.
f7f619c: "perf(ascend): accelerate FM9G70 paged serving"This commit contains four related performance changes.
3.1 Build integration
src/native/ascend/custom/CMakeLists.txtpaged_attention.cppandpaged_attention_fai.cpptono_workspace_kernel.src/CMakeLists.txtINFINILM_ENABLE_CUSTOM_PAGED_ATTENTION=1when the Ascend custom kernel library is built.3.2 CANN FAInferAttentionScore kernel infrastructure
Files under
src/native/ascend/custom/fused_infer_attention_score/import the dependency-closed AscendC subset required by the FM9G70 BF16 decode kernel. A build-dependency audit retained 45 files used bypaged_attention_fai.cpp; 16 upstream files for broader regular-prefill, MLA, tiling, and unused tile utilities were removed.attn_infra/arch/*attn_infra/coord.hpp,layout/*, andmatrix_coord.hppattn_infra/gemm/*attn_infra/epilogue/*flash_attention_regular_decode.handkernel_common.hppIntent: provide a fused BF16 MHA paged-decode kernel that keeps QK/PV math, online softmax, and output rescaling on device instead of decomposing attention into smaller operators.
3.3 Paged attention wrappers
src/native/ascend/custom/paged_attention/op_kernel/paged_attention_fai.cppsrc/native/ascend/custom/paged_attention/op_kernel/paged_attention.cppsrc/native/ascend/ops/flash_attn_with_kvcache/kernel.hInfinilmFaiTilingdevice allocation for the specialized decode path.RunFai.3.4 Faster paged cache writes
src/native/ascend/custom/reshape_and_cache_flash/op_kernel/reshape_and_cache_flash.cppsrc/native/ascend/ops/reshape_and_cache_flash/kernel.hIntent: remove the single-block serialization bottleneck and bound each cache-write launch. Profiling previously showed this operator consuming about 36.8% of per-rank device time; parallel work distribution turns the launch into a bounded, high-occupancy cache write.
3.5 Uniform prefill metadata
src/native/ascend/ops/flash_attn_varlen_func/kernel.haclrtMemcpyAsyncand stream synchronization.innerPrecise=0setting for the FM9G70 Ascend TND paged-prefill path.Intent: remove avoidable host synchronization from fixed-shape serving prefill and use the precision mode already validated for this model/path.
Type of Change
feat— new feature / new operator / new platformperf— performance improvementfix— bug fixrefactor— code restructuring without behavior changetest— adding or fixing tests onlydocs— documentation onlybuild— build system changeschore— tooling, formatting, or other non-code changesPlatforms Affected
WITH_CPU)WITH_NVIDIA)WITH_ILUVATAR)WITH_METAX)WITH_CAMBRICON)WITH_MOORE)WITH_ASCEND)WITH_TORCH)Smoke Test Result
Built against CANN 9.0.1 on Ascend 910B3 with
WITH_ASCEND=ONand the pinned InfiniRT prefix:The full default
alltarget is currently blocked by the upstreamexamples/tensorandexamples/gemmlink failure forinfini::rt::TensorView::ToString()in this runtime installation. The operator library itself, its install target, and dependent InfiniLM links all build successfully.Test Results on Supported Platforms
infiniopstarget, install, dependent InfiniLM link/importinfiniPython bindingNumerical validation
The existing FP16 GQA fallback numerical failure is unchanged upstream behavior and is not used by the FM9G70B BF16 MHA path.
Benchmark / Performance Impact
End-to-end validation used FM9G70B BF16 MHA, TP=8, Ascend 910B3, CANN 9.0.1, paged attention, FlashAttention, graph capture, pre-transpose, 192 KV blocks, 256-token block size, and max decode batch 16.
These numbers validate the complete dependent InfiniOps/InfiniCore/InfiniLM stack, not this repository in isolation:
Across all 33 cases with a valid Jingbo baseline:
The final matrix completed all 36 requested cases. The remaining three
c64/in1024cases had no valid Jingbo baseline. After the dependency prune, a clean-stack regression onc64/in32/out256produced 454.77 output tok/s, 221.79 ms mean TTFT, and 127.36 ms mean TPOT, remaining ahead of the Jingbo baseline for that case.Profiling motivation:
Notes for Reviewers
The FAI directory contains Huawei-authored code marked under CANN Open Software License Agreement Version 2.0. The public
ascend/cann-opsrepository currently publishes a license file titled Version 1.0; that exact upstream text is retained atLICENSES/CANN-OSL-upstream.txt. Obtaining the authoritative 2.0 text or an upstream correction is a merge prerequisite.The NZ operators intentionally reject unsupported strides, dtypes, and non-16-aligned shapes rather than attempting silent fallback.
MatmulWeightNzretains ACL descriptors for repeatability and rebinds raw addresses on subsequent calls; reviewers may want to focus on lifetime versus ACL runtime shutdown.The FAI path is deliberately guarded to BF16 MHA with contiguous paged KV, block table, and cache lengths. FP16/GQA and non-contiguous requests retain the existing fallbacks.
The cache-write kernel has a fixed maximum of 64 parallel blocks and 4096 tokens per launch to bound workspace and maintain graph compatibility.
The benchmark numbers are end-to-end InfiniLM results and include dependent InfiniCore/InfiniLM integration commits; CI should still run the repository-level operator tests on a runtime with Python bindings installed.