ggml: CONCAT backward slices a contiguous gradient (a transposed one read the wrong elements)

[joelteply]

The bug. CONCAT's backward slices the gradient with ggml_view_4d(grad, …, grad->nb[1..3], offset), and a view's first stride is always the element size. The training attention transposes V for its KQV product (v = ggml_cont(ggml_transpose(v)) in build_attn_mha). So V's gradient reaches the walk's concat ([cached prefix | this chunk's V]) through a transpose, with a first stride that is not the element size, and the slice reads the right number of elements from the wrong places. K is only permute(0,2,1,3), which keeps the first stride, so K came through intact.

Measured (1.5B Q4_K_M on the 5090, test-walk-exact on the walk branch, with the SGD step read off the adapter, so step = −lr·g):

• The exact walk's cross-chunk V term had the right norm (0.93) and was orthogonal to the true one (cosine −0.10). K's was aligned.
• With the fix, the exact walk in 4 chunks matches one graph at cosine 0.998, and the plain walk's step against one graph moves from cosine 0.54 to 0.84.

So every walk-trained adapter has had its attn_v gradient scrambled on every chunk that had a cached prefix. Any other concat whose gradient arrives non-contiguous was affected the same way.

The fix: slice a contiguous gradient (ggml_cont only when it isn't one already). It's one commit, standalone on the production line, and test-walk-exact mutation-checks it on #40's follow-up (it fails at cosine 0.43 without it).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc

[joelteply]

Review at fbd9ba5 (Fable). The fix is right: ggml_view_4d takes nb[1..3] only and assumes nb[0] is the element size, so a gradient that arrives through a transpose (V in the training attention) was sliced from the wrong addresses. Slicing a contiguous copy, with offsets from g->nb[dim], is the correct fix. ggml_is_contiguous also routes a permuted or gapped gradient through the copy, which is what we want.

One ask before merge: a regression test that fails without this hunk. This is the silent-wrong-gradient class we keep missing. Only your cosine measurement caught it, and nothing in CI would catch it if it came back. test-backend-ops' gradient mode already checks CONCAT, but with a contiguous consumer, so it passes either way. A case whose CONCAT output is consumed through a transpose (e.g. concat on dim 1, then ggml_cont(ggml_transpose(...)), then the loss), so the gradient reaches CONCAT non-contiguous, would gate it. Please confirm it fails with the ggml.c hunk reverted. Approve once that's in.

[joelteply]

Approve at b8c1f93 (Fable). Verified on the M5 (CPU build of this head): test-backend-ops grad -p ne_b_d gives 193/193 with the fix. With the ggml.c hunk reverted, the new case fails with CONCAT(ne_a=[4,3,5,1],ne_b_d=2): [CONCAT] MAA = 13.756570002 > 0.000100000 FAIL, and every other CONCAT case still passes, so the new case is the only gate on this bug. The SQR trick is the right call: a SUM's all-ones gradient survives any scramble.

[joelteply]

Done (b8c1f93): test_concat_grad_transposed in test-backend-ops. CONCAT's output reaches the loss through permute(0,2,1,3) then a transpose, the way the training attention consumes the walk's V, so the gradient arrives with a first stride that isn't the element size. The output is squared, because the harness differentiates a SUM (all-ones gradient) and a scrambled tensor of ones is still ones; my first version without the square passed with the fix reverted, so it caught nothing. Now, with the fix reverted: [CONCAT] MAA = 13.548662675 > 0.0001 FAIL; with it, OK. (test-backend-ops grad segfaults for every op on this Windows CPU build, ADD included after 51 passing cases, so I ran it filtered with -p; that's a harness problem here, separate from this.)