Repository navigation
fix: skip zero-variance GRPO updates - #1118
Merged
Merged
Conversation
bradhilton
force-pushed
the
tarkin/grpo-zero-variance-advantages
branch
2 times, most recently
from
October 10, 2026 17:36
a234491 to
ad8f435
Compare
bradhilton
force-pushed
the
tarkin/grpo-zero-variance-advantages
branch
from
October 10, 2026 18:58
ad8f435 to
b66e9ff
Compare
bradhilton
marked this pull request as ready for review
October 10, 2026 19:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent: Tarkin
Fixes #1110.
Problem: GRPO advantages are
(r − mean) / (std + 1e-6), and a trajectory is dropped only when its advantage is exactly 0. In a group with equal rewards, floating-point residue in the mean can leave a tiny nonzero advantage:[0.1, 0.1, 0.1]gives −1.4e-11. Those trajectories were kept.pack.pythen divided by the batch-mean |advantage| with no epsilon. So a batch made only of such groups trained on |A| = 1 for every token, all with the same sign, and a batch of exact zeros produced NaN.Behavior after this change:
art.utils.grpo.rewards_have_zero_variance, shared by tokenization, the pipeline trainer and thedata/step_num_groups_trainablemetric.pack.pyonly normalizes weights and advantages when their means are positive, so it can no longer produce NaN or Inf.Validation:
tests/unit/test_grpo_zero_variance.pycovers residue, exact-zero and near-equal groups, with scaling on and off and dropping on and off. It checks that normal and mixed batches are unchanged, that zero-weight denominators stay finite, and that the Local, Tinker and Megatron backends skip cleanly with unchanged optimizer state, including a four-rank no-op. 31 of its 50 cases fail on main.Limitations:
tinker_nativebuilds advantages separately and has its own equal-reward residue issue. It isn't touched here; see fix: use population std for native Tinker advantages #1115 and Tracking: numerical issues found in an audit of ART main after #1105 #1111.