Skip to content

fix(embedding): initialize untied vocabulary parallel weights - #269

Open
superme1one wants to merge 1 commit into
InfiniTensor:masterfrom
superme1one:fix/initialize-vocab-parallel-embedding
Open

superme1one wants to merge 1 commit into
InfiniTensor:masterfrom
superme1one:fix/initialize-vocab-parallel-embedding

Conversation

@superme1one

@superme1one superme1one commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

Initialize the local weight shard of an untied VocabParallelEmbedding when the module is constructed.

Motivation

The same-version control left the embedding allocation uninitialized; two of three runs produced weights with absolute values up to 3.03036e35.

Type of Change

fix; no public API removal.

Platforms Affected

Tensor-parallel module construction; CPU and CUDA.

Smoke Test Result

Three same-seed candidate initializations produced the same 4,096 weights, mean -0.00027719 and standard deviation 1.00277663. Tied/untied forwards passed and CPU-to-GPU weights matched exactly. Initialization, requires-grad and device migration regressions passed. These checks used TP=1; multi-rank initialization still needs distributed CI.

The combined candidate stack passed 228 CPU tests (3 skips) and 229 Iluvatar GPU tests (2 CPU-only death checks skipped), with no failures. Coverage includes optimizer state, shared parameters, autograd and checkpoint round trips. The GPU build uses CoreX 4.3.0.20250624 with a separately recorded compatibility layer. NVIDIA execution and hosted CI are pending.

Test Results on Supported Platforms

Validated on the platforms listed above. Local formatting and patch-application checks passed. Hosted CI is tracked separately from the recorded target-platform tests. Affected backends not available in this session require their normal CI/owner checks.

Benchmark / Performance Impact

Correctness fix; no performance gain claimed.

Notes for Reviewers

Base: InfiniTensor/InfiniTrain@e19a56dcb56bec11f1af63c089a1e8ae03ff24c7. Verification date: 2026-11-16. Detailed test inputs, source hashes and logs are retained in the separate validation package; performance reports are excluded from the source patch.

Closes #265.

CI approval status at submission

The upstream pull-request workflows are action_required and have not scheduled jobs. A repository maintainer needs to approve the runs. Target-platform tests are recorded above. This repository did not automatically run checks for the feature-branch push. Local formatting and target-platform results are recorded above. Backend tests requiring upstream infrastructure remain pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VocabParallelEmbedding leaves untied weights uninitialized

1 participant