Skip to content

fix(autograd): schedule backward over the root-reachable graph - #276

Open
superme1one wants to merge 2 commits into
InfiniTensor:masterfrom
superme1one:codex/fix-backward-reachable-graph
Open

superme1one wants to merge 2 commits into
InfiniTensor:masterfrom
superme1one:codex/fix-backward-reachable-graph

Conversation

@superme1one

Copy link
Copy Markdown

Backward currently waits for forward consumers outside the selected loss and for every differentiable output of a multi-output function. Creating an unused branch, or selecting only one Split output, can therefore leave leaf gradients null.

Fixes #273.

  • Count incoming edges only within the graph reachable from the selected backward root. Keep counters and partial gradients in a per-call context so reentrant checkpoint backward cannot overwrite pending outer gradients.
  • Save shape/dtype/device metadata for differentiable multi-output results and materialize zero gradients for unused outputs. Non-differentiable outputs remain null.
  • Propagate completed null-gradient edges so they do not block a shared predecessor; release a leaf accumulator even when its gradient is null. Preserve the existing leaf-first recursive traversal.

Ten device-parameterized regression tests cover live/destroyed unused branches, shared intermediates and repeated edges, uneven Split outputs, all-used output accumulation, null-gradient propagation and lifetime cleanup, non-differentiable TopK indices, both checkpoint modes, and pending shared-parameter gradients during reentrant backward. The original branch/Split cases fail on unmodified master.

Validation on Linux x86_64, CPU Release, Clang 18.1.8/GCC 13.4.0 libstdc++, CMake 3.31.6, with CUDA/NCCL/OpenMP disabled:

Runtime validation in this submission was CPU-only. Upstream CI requires maintainer approval for this contributor; the fork results use the same workflow definitions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Autograd drops gradients when unused branches or Split outputs exist

1 participant