Skip to content

fix(layernorm): center FP32 statistics before variance reduction - #270

Open
superme1one wants to merge 1 commit into
InfiniTensor:masterfrom
superme1one:fix/fp32-layernorm-variance
Open

superme1one wants to merge 1 commit into
InfiniTensor:masterfrom
superme1one:fix/fp32-layernorm-variance

Conversation

@superme1one

@superme1one superme1one commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

Center FP32 LayerNorm statistics before reducing variance and use the centered values consistently in backward.

Motivation

Large means with small variance caused cancellation and non-finite results in the original FP32 implementation.

Type of Change

fix; no public API removal.

Platforms Affected

CUDA FP32 LayerNorm, tested through the Iluvatar compatibility build.

Smoke Test Result

Forward, input gradient, weight gradient and bias gradient passed all 60 checks for 15 conditioning cases and all 792 checks for 198 boundary cases against FP64 references computed from the exact same FP32 inputs. The original control failed 32/60 stress checks and produced 15,882 non-finite checked values. Against independent PyTorch FP32 references, the candidate differs in 12/60 stress and 190/792 extended checks; that FP32 reference also differs from the FP64 oracle in the same number of checks. Tolerances were preserved.

The combined candidate stack passed 228 CPU tests (3 skips) and 229 Iluvatar GPU tests (2 CPU-only death checks skipped), with no failures. Coverage includes optimizer state, shared parameters, autograd and checkpoint round trips. The GPU build uses CoreX 4.3.0.20250624 with a separately recorded compatibility layer. NVIDIA execution and hosted CI are pending.

Test Results on Supported Platforms

Validated on the platforms listed above. Local formatting and patch-application checks passed. Hosted CI is tracked separately from the recorded target-platform tests. Affected backends not available in this session require their normal CI/owner checks.

Benchmark / Performance Impact

Numerical-stability fix. This is not a bitwise-equivalence claim.

Notes for Reviewers

Base: InfiniTensor/InfiniTrain@e19a56dcb56bec11f1af63c089a1e8ae03ff24c7. Verification date: 2026-11-16. Detailed test inputs, source hashes and logs are retained in the separate validation package; performance reports are excluded from the source patch.

Closes #266.

CI approval status at submission

The upstream pull-request workflows are action_required and have not scheduled jobs. A repository maintainer needs to approve the runs. Target-platform tests are recorded above. This repository did not automatically run checks for the feature-branch push. Local formatting and target-platform results are recorded above. Backend tests requiring upstream infrastructure remain pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FP32 LayerNorm variance cancellation produces non-finite outputs

1 participant