Skip to content

feat(ascend): accelerate FM9G70 paged serving - #1041

Open
wooway777 wants to merge 3 commits into
masterfrom
merge/fm9g70-ascend-stack
Open

wooway777 wants to merge 3 commits into
masterfrom
merge/fm9g70-ascend-stack

Conversation

@wooway777

@wooway777 wooway777 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR ports and productizes the FM9G70 Ascend serving stack. It contains three intentionally ordered commits:

  1. Add Ascend FRACTAL_NZ dense-weight conversion and matmul operators.
  2. Mark the FM9G hot operators as graph-capture safe and make the NZ matmul executor repeatable.
  3. Add the custom CANN FAI paged-attention decode path and accelerate paged cache writes and uniform prefill.

The change is Ascend-only. Existing generic and non-Ascend paths remain unchanged, and unsupported Ascend layouts fall back to the previous implementations.

Motivation

FM9G70B BF16 MHA serving on Ascend 910B3 was bottlenecked by four separate costs:

  • Dense weights need Huawei FRACTAL_NZ layout for efficient Ascend matmul, but the public operator surface previously exposed no NZ conversion/matmul pair.
  • Hot serving operators were conservatively excluded from device graph capture, forcing extra graph segmentation.
  • Paged decode attention spent most of its time in the generic ACLNN path.
  • reshape_and_cache_flash_strided launched one serial work block for every token/head and consumed about 36.8% of device time per rank (average 7.55 ms in the profiled serving shape).

This PR supplies the missing layout operators and replaces the hot FM9G70 paged decode/cache-write paths with specialized Ascend implementations while retaining guarded fallbacks.

Change Intent by Commit

1. f91d31f1: "feat(ascend): add FRACTAL_NZ weight layout support"

Intent: provide an explicit, shape-checked operator pair for converting and consuming FRACTAL_NZ dense weights.

  • src/base/matmul_weight_nz.h

    • Adds the operator contract MatmulWeightNz(input, weight, out).
    • Restricts the public contract to 2D FP16/BF16 tensors.
    • Requires contiguous input, weight, and output and 16-aligned weight dimensions.
    • These assertions make ACLNN FRACTAL_NZ alignment requirements explicit instead of allowing an opaque ACLNN failure later.
  • src/base/trans_matmul_weight_nz.h

    • Adds TransMatmulWeightNz(input, output).
    • Validates matching 2D shapes, FP16/BF16 dtype, contiguity, and 16 alignment.
    • This operator represents a layout conversion into the NZ-compatible representation.
  • src/native/ascend/ops/trans_matmul_weight_nz/kernel.h

    • Implements the Ascend backend with aclnnTransMatmulWeightNz.
    • Reuses the normal workspace-pool and stream execution conventions.
    • Intent: perform weight conversion on-device and avoid host-side transposes or generic transpose kernels.
  • src/native/ascend/ops/matmul_weight_nz/kernel.h

    • Implements MatmulWeightNz with aclnnMatmulWeightNz.
    • Creates an ACL tensor in ACL_FORMAT_FRACTAL_NZ for the converted weight while keeping input/output descriptors contiguous.
    • Intent: let the ACLNN matmul consume the preferred Ascend weight layout directly.
  • tests/test_matmul_weight_nz.py

    • Adds FP16/BF16 numerical coverage for three input/weight shapes.
    • Checks the combined conversion-plus-matmul result against torch.matmul.
    • The original external contribution contained syntactically invalid compressed Python; this branch reformats it into valid pytest source while preserving the original author.

2. 3ad9b36e: "perf(ascend): mark FM9G hot kernels graph-capture safe"

Intent: reduce artificial graph segmentation for the FM9G70 capture-enabled serving graph.

  • The following wrappers now override capture_safe() to true:

    • add
    • argmax
    • copy
    • embedding
    • fill
    • flash_attn_varlen_func
    • flash_attn_with_kvcache
    • fused_add_rms_norm
    • gemm
    • linear
    • matmul
    • matmul_weight_nz
    • mul
    • reshape_and_cache_flash
    • rms_norm
    • rotary_embedding
    • silu_and_mul
  • These operators launch through ACLNN or the existing generated launchers on the caller's stream and do not require host-side work between capture and replay. Marking them capture-safe allows larger device graph segments and reduces per-layer graph boundaries.

  • src/native/ascend/ops/embedding/kernel.h also contains formatting-only adjustment around a deprecated constructor; there is no behavior change beyond capture_safe().

  • src/native/ascend/ops/matmul_weight_nz/kernel.h is changed more substantially:

    • ACL tensor descriptors and the aclOpExecutor are retained after the first invocation.
    • aclSetAclOpExecutorRepeatable marks the executor reusable.
    • Subsequent calls call aclSetRawTensorAddr to bind new input/weight/output addresses instead of recreating descriptors.
    • Workspace size is cached.
    • Intent: eliminate descriptor/workspace setup inside graph capture/replay and ensure repeated calls use stable ACLNN state.
    • The retained descriptors intentionally follow the existing AclTensorCache lifetime convention.

3. f7f619c: "perf(ascend): accelerate FM9G70 paged serving"

This commit contains four related performance changes.

3.1 Build integration

  • src/native/ascend/custom/CMakeLists.txt
    • Adds paged_attention.cpp and paged_attention_fai.cpp to no_workspace_kernel.
  • src/CMakeLists.txt
    • Defines INFINILM_ENABLE_CUSTOM_PAGED_ATTENTION=1 when the Ascend custom kernel library is built.
    • Intent: compile the new paged-attention launchers and make their generated launch headers available to the provider wrappers.

3.2 CANN FAInferAttentionScore kernel infrastructure

Files under src/native/ascend/custom/fused_infer_attention_score/ import the dependency-closed AscendC subset required by the FM9G70 BF16 decode kernel. A build-dependency audit retained 45 files used by paged_attention_fai.cpp; 16 upstream files for broader regular-prefill, MLA, tiling, and unused tile utilities were removed.

  • attn_infra/arch/*
    • Provides architecture-level resource management, local tensor buffers, and cross-core synchronization primitives.
  • attn_infra/coord.hpp, layout/*, and matrix_coord.hpp
    • Models block coordinates, matrix/vector layouts, and tile coordinates used by the split attention schedule.
  • attn_infra/gemm/*
    • Implements QK and PV MMAD tiles, including decode-specific tiles.
    • Provides L1/L0A/L0B/UB copy helpers and dispatch policies.
  • attn_infra/epilogue/*
    • Implements online-softmax initialization, rescaling, low-precision output paths, and output copies.
    • The tile utilities cover broadcasting, casts, elementwise multiply/add, and UB/GM movement.
  • flash_attention_regular_decode.h and kernel_common.hpp
    • Provide the decode kernel entry and shared host/device definitions used by the generated launcher.

Intent: provide a fused BF16 MHA paged-decode kernel that keeps QK/PV math, online softmax, and output rescaling on device instead of decomposing attention into smaller operators.

3.3 Paged attention wrappers

  • src/native/ascend/custom/paged_attention/op_kernel/paged_attention_fai.cpp
    • Adds the FAI-based BF16 paged-attention kernel entry.
  • src/native/ascend/custom/paged_attention/op_kernel/paged_attention.cpp
    • Adds the accompanying custom paged-attention launch path.
  • src/native/ascend/ops/flash_attn_with_kvcache/kernel.h
    • Adds an InfinilmFaiTiling device allocation for the specialized decode path.
    • Adds sequence-dimension view squeezing for decode tensors whose sequence dimension is one.
    • Adds strict eligibility checks for BF16 MHA, matching head counts, contiguous KV/cache/block-table/length tensors, and compatible strides.
    • Routes eligible FM9G70 BF16 MHA decode requests to RunFai.
    • Retains the previous custom paged path and ACLNN fallback for FP16, GQA, non-contiguous tensors, scalar cache lengths, and unsupported layouts.
    • Copies tiling/data in bounded segments rather than requiring one very large contiguous staging allocation.
    • Intent: use the fast path only where its assumptions are proven, without changing behavior for existing clients.

3.4 Faster paged cache writes

  • src/native/ascend/custom/reshape_and_cache_flash/op_kernel/reshape_and_cache_flash.cpp
    • Updates the generated strided-cache kernel so one launch can process multiple token/head work items.
  • src/native/ascend/ops/reshape_and_cache_flash/kernel.h
    • Raises parallelism to at most 64 cache-write work blocks.
    • Splits large prefill batches into chunks of at most 4096 tokens.
    • Advances key/value/slot pointers by token offset while preserving all stride parameters.

Intent: remove the single-block serialization bottleneck and bound each cache-write launch. Profiling previously showed this operator consuming about 36.8% of per-rank device time; parallel work distribution turns the launch into a bounded, high-occupancy cache write.

3.5 Uniform prefill metadata

  • src/native/ascend/ops/flash_attn_varlen_func/kernel.h
    • Detects equal-length prompts using sequence count, maximum Q/K lengths, and total token count.
    • Builds Q/K cumulative length arrays on host for that proven-uniform case.
    • Avoids a per-layer device-to-host aclrtMemcpyAsync and stream synchronization.
    • Uses the validated high-throughput innerPrecise=0 setting for the FM9G70 Ascend TND paged-prefill path.
    • Corrects negative next-token handling so a negative window bound maps to the actual maximum token count.

Intent: remove avoidable host synchronization from fixed-shape serving prefill and use the precision mode already validated for this model/path.

Type of Change

  • feat — new feature / new operator / new platform
  • perf — performance improvement
  • fix — bug fix
  • refactor — code restructuring without behavior change
  • test — adding or fixing tests only
  • docs — documentation only
  • build — build system changes
  • chore — tooling, formatting, or other non-code changes
  • Breaking change

Platforms Affected

  • CPU ( WITH_CPU)
  • NVIDIA ( WITH_NVIDIA)
  • Iluvatar ( WITH_ILUVATAR)
  • MetaX ( WITH_METAX)
  • Cambricon ( WITH_CAMBRICON)
  • Moore ( WITH_MOORE)
  • Ascend ( WITH_ASCEND)
  • PyTorch C++ bindings ( WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

Smoke Test Result

Built against CANN 9.0.1 on Ascend 910B3 with WITH_ASCEND=ON and the pinned InfiniRT prefix:

cmake --build .../InfiniOps-abi0b --target infiniops --parallel 8
[100%] Built target infiniops

cmake --install .../InfiniOps-abi0b
-- Install configuration: "Release"
-- Up-to-date / install completed

python -m py_compile tests/test_matmul_weight_nz.py
# no output; exit 0

The full default all target is currently blocked by the upstream examples/tensor and examples/gemm link failure for infini::rt::TensorView::ToString() in this runtime installation. The operator library itself, its install target, and dependent InfiniLM links all build successfully.

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
Ascend Yes Pass: infiniops target, install, dependent InfiniLM link/import Numerical checks below; NZ pytest requires a runtime with the infini Python binding
CPU No N/A Base contracts are backend-independent but no CPU backend is added
NVIDIA / Iluvatar / MetaX / Cambricon / Moore No N/A No backend implementation changed

Numerical validation

BF16 MHA FAI vs reference:
max abs error = 0.001953125
test tolerance = 2e-2
result = passed

reshape_and_cache_flash numerical tests:
result = passed

tests/test_matmul_weight_nz.py:
Python syntax/compile = passed
runtime pytest = not run in this container:  `infini` Python binding is not installed

The existing FP16 GQA fallback numerical failure is unchanged upstream behavior and is not used by the FM9G70B BF16 MHA path.

Benchmark / Performance Impact

End-to-end validation used FM9G70B BF16 MHA, TP=8, Ascend 910B3, CANN 9.0.1, paged attention, FlashAttention, graph capture, pre-transpose, 192 KV blocks, 256-token block size, and max decode batch 16.

These numbers validate the complete dependent InfiniOps/InfiniCore/InfiniLM stack, not this repository in isolation:

Concurrency Output throughput vs Jingbo baseline
c1 +12.3% to +13.1%
c4 +6.5% to +10.9%
c16 +4.1% to +8.9%
c64 +3.2% to +22.0%

Across all 33 cases with a valid Jingbo baseline:

  • output throughput: 33 wins / 0 losses
  • mean TPOT: 33 wins / 0 losses
  • mean TTFT: 33 wins / 0 losses

The final matrix completed all 36 requested cases. The remaining three c64/in1024 cases had no valid Jingbo baseline. After the dependency prune, a clean-stack regression on c64/in32/out256 produced 454.77 output tok/s, 221.79 ms mean TTFT, and 127.36 ms mean TPOT, remaining ahead of the Jingbo baseline for that case.

Profiling motivation:

reshape_and_cache_flash_strided before optimization:
~36.8% device time per profiled rank
average duration ~7.55 ms

Notes for Reviewers

  • The FAI directory contains Huawei-authored code marked under CANN Open Software License Agreement Version 2.0. The public ascend/cann-ops repository currently publishes a license file titled Version 1.0; that exact upstream text is retained at LICENSES/CANN-OSL-upstream.txt. Obtaining the authoritative 2.0 text or an upstream correction is a merge prerequisite.

  • The NZ operators intentionally reject unsupported strides, dtypes, and non-16-aligned shapes rather than attempting silent fallback.

  • MatmulWeightNz retains ACL descriptors for repeatability and rebinds raw addresses on subsequent calls; reviewers may want to focus on lifetime versus ACL runtime shutdown.

  • The FAI path is deliberately guarded to BF16 MHA with contiguous paged KV, block table, and cache lengths. FP16/GQA and non-contiguous requests retain the existing fallbacks.

  • The cache-write kernel has a fixed maximum of 64 parallel blocks and 4096 tokens per launch to bound workspace and maintain graph compatibility.

  • The benchmark numbers are end-to-end InfiniLM results and include dependent InfiniCore/InfiniLM integration commits; CI should still run the repository-level operator tests on a runtime with Python bindings installed.

Jingbo-gao and others added 2 commits November 13, 2026 23:48
Convert eligible weights with aclnnTransMatmulWeight and execute them through aclnnMatmulWeightNz.
Cache the repeatable MatmulWeightNz executor and qualify the hot Ascend decode kernels for device graph capture. This ports the capture behavior used by the FM9G70B Ascend stack to the current operator API.
@wooway777
wooway777 requested a review from a team November 14, 2026 03:35
@wooway777 wooway777 changed the title Merge/fm9g70 ascend stack feat(ascend): accelerate FM9G70 paged serving Nov 14, 2026
@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch from 374fa17 to b956fdd Compare November 16, 2026 07:41
Port the CANN FAI decode path, parallelize and chunk paged cache writes, and retain host-side uniform prefill sequence metadata to reduce synchronous D2H copies.
@wooway777
wooway777 force-pushed the merge/fm9g70-ascend-stack branch from b956fdd to f7f619c Compare November 16, 2026 08:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants