Skip to content

perf(activation): fuse FP32 NewGELU forward and backward - #271

Open
superme1one wants to merge 2 commits into
InfiniTensor:masterfrom
superme1one:perf/fp32-newgelu-fusion
Open

superme1one wants to merge 2 commits into
InfiniTensor:masterfrom
superme1one:perf/fp32-newgelu-fusion

Conversation

@superme1one

@superme1one superme1one commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

Fuse contiguous FP32 NewGELU forward and backward, saving the tanh intermediate for autograd and retaining the existing expression for other dtypes and layouts.

Motivation

The unfused activation launches several elementwise operations and materializes intermediate tensors.

Type of Change

perf; no public API removal.

Platforms Affected

CUDA FP32 NewGELU implementation; Iluvatar CoreX port executed.

Smoke Test Result

Current-interface forward/backward tests and unfused fallback tests passed. GPT-2 124M, batch 1, sequence 64: a 10,240-step FP32 candidate/control comparison passed the predefined loss threshold of 0.001 (maximum difference 0.000036239) and sampled final parameter allclose with rtol=1e-4, atol=1e-5 (maximum difference 3.58e-7). Parameters remained finite and tied weights retained shared identity. The initial control process had a non-reproducible loss excursion; that failed record is retained and excluded from performance measurements.

The combined candidate stack passed 228 CPU tests (3 skips) and 229 Iluvatar GPU tests (2 CPU-only death checks skipped), with no failures. Coverage includes optimizer state, shared parameters, autograd and checkpoint round trips. The GPU build uses CoreX 4.3.0.20250624 with a separately recorded compatibility layer. NVIDIA execution and hosted CI are pending.

Test Results on Supported Platforms

Validated on the platforms listed above. Local formatting and patch-application checks passed. Hosted CI is tracked separately from the recorded target-platform tests. Affected backends not available in this session require their normal CI/owner checks.

Benchmark / Performance Impact

GPT-2 124M, FP32, batch 1, sequence 64, 512 steps per process, three alternating control/candidate pairs on TG-V200. Median training-loop wall time excluding model loading changed from 32.08727 s to 29.96560 s, a 6.61% decrease. Every pair passed loss and sampled-parameter comparisons. Both builds share all other candidate changes; only NewGELU fusion differs. The historical 19.18% timing reduction is not used for this revision.

Notes for Reviewers

Base: InfiniTensor/InfiniTrain@e19a56dcb56bec11f1af63c089a1e8ae03ff24c7. Verification date: 2026-11-16. Detailed test inputs, source hashes and logs are retained in the separate validation package; performance reports are excluded from the source patch.

CI approval status at submission

The upstream pull-request workflows are action_required and have not scheduled jobs. A repository maintainer needs to approve the runs. Target-platform tests are recorded above. This repository did not automatically run checks for the feature-branch push. Local formatting and target-platform results are recorded above. Backend tests requiring upstream infrastructure remain pending.

Validation after include cleanup

Added the datatype/device and standard-library includes used by NewGELU and its regression. Calculation behavior is unchanged.

  • Commit: b76ca09bcfa91a6dcfafe1d5d703400d2a56e9b5.
  • Clean CPU Release build on the Iluvatar server: 146 passed; 3 existing device-dependent autograd tests skipped (autograd and module suites).
  • Full, unchanged fork workflows passed: Format Check, clang-tidy, IWYU.
  • The new upstream workflows are awaiting maintainer approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant