Skip to content

fix: preserve FP32 target gradients in Megatron TE cross entropy - #1117

Merged
bradhilton merged 2 commits into
mainfrom
tarkin/te-ce-fp32-target-grad
Oct 12, 2026
Merged

bradhilton merged 2 commits into
mainfrom
tarkin/te-ce-fp32-target-grad

Conversation

@bradhilton

@bradhilton bradhilton commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1108.

Problem: ART's Megatron backend uses Transformer Engine's fused cross-entropy (cross_entropy_fusion_impl = "te"). TE's Triton kernel stores bf16(p) for every vocabulary entry, then reloads the target entry and subtracts 1 in BF16. The target gradient is therefore exactly 0 whenever p ≥ 0.998, the same dead zone #1105 fixed in TrainerRank. In a GPU sweep through Megatron's loss with ART's settings, 294 of 512 rows had a zero target gradient. A real ART SFT step with near-certain targets had 163 of 216.

Behavior after this change:

  • art.megatron.te_cross_entropy patches TE's kernel source at import, alongside ART's existing bridge patches.
  • The target term is now subtracted while the gradient is still FP32, and the old reload-and-subtract step is gone. The kernel still does a single store in the logits dtype.
  • Forward losses are bitwise unchanged.
  • The patch requires an exact match of the TE 2.11/2.14 source it replaces. On any other version it raises at import, rather than silently keeping the bug.

Validation:

  • GPU sweep (H200): zero target gradients drop from 294/512 to 0/512, and target-gradient error now matches Megatron's own FP32 paths. This ran on the installed TE 2.11. TE 2.14.1's kernels and wrapper were also loaded into the same process and passed the regression and TP/CP checks.
  • Real ART SFT step (tiny Llama, rank-8 LoRA, near-certain targets): zero target gradients drop from 163/216 to 0/216. LoRA-gradient cosine against the FP32 reference goes from 0.9646 to 0.999999, and the loss is bitwise identical.
  • Cost of the cross-entropy step (vocab 152k): peak memory is identical, and time rises 1–4%. Megatron's FP32 alternatives (native or unfused) would need 4–6× the memory (14 GB versus 2.4 GB at 8,192 rows) and be 3–5× slower.
  • New tests: CPU tests cover the patched gradient arithmetic, idempotence and refusal of unknown source. A CUDA test compares against FP64 on GPU machines.

Limitations:

  • Upgrading TE to a version with a different kernel source will fail at import until this patch is updated.
  • The patch relies on two private Triton APIs: _unsafe_update_src, plus clearing the kernel's compiled cache.
  • TE 2.14.1 ran only inside the CUDA 12 environment, not a full CUDA 13 stack.
  • TE still stores the gradient in BF16 before per-token RL weights are applied. That's a separate, much smaller precision effect.

@bradhilton
bradhilton force-pushed the tarkin/te-ce-fp32-target-grad branch 3 times, most recently from 05b87f0 to a29149c Compare October 10, 2026 18:58
@bradhilton
bradhilton marked this pull request as ready for review October 10, 2026 19:26
@bradhilton
bradhilton requested a review from FurtherAI October 10, 2026 19:34
@bradhilton
bradhilton force-pushed the tarkin/te-ce-fp32-target-grad branch from a29149c to c52869c Compare October 10, 2026 19:38
@bradhilton
bradhilton merged commit c7dda22 into main Oct 12, 2026
8 checks passed
@bradhilton
bradhilton deleted the tarkin/te-ce-fp32-target-grad branch October 12, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Megatron backend: Transformer Engine fused cross-entropy zeroes target gradients for p ≥ 0.998 (same bug as #1105)

2 participants