Skip to content

fix(models): never treat a truncated weight download as complete - #465

Merged
lightningpixel merged 1 commit into
devfrom
fix/incomplete-model-weights
Nov 18, 2026
Merged

lightningpixel merged 1 commit into
devfrom
fix/incomplete-model-weights

Conversation

@lightningpixel

Copy link
Copy Markdown
Owner

Refs #389

TripoSG failed to load with SafetensorError: Error while deserializing header: incomplete metadata, file not fully covered: a weight file on disk was truncated, yet Modly reported the model as downloaded, and every Generate showed a raw traceback.

Root cause

_download_file_streamed (api/routers/model.py) treated an early end of stream as success. When the server closed the connection early, http.client returned b"" instead of raising, and the short .part file was renamed into place. After that, any non-empty final file was skipped, so downloading again never repaired it.

Changes

  • Expected sizes: both download routes list the repo with HfApi.list_repo_tree (one call, sizes included) instead of list_repo_files.
  • Size check: each download is checked against the expected size, or the response's Content-Length/Content-Range when no size is known.
    • A short file raises IncompleteDownload. The existing retry loop then resumes from the .part with a Range request.
    • A short file is never renamed into place.
    • An existing final file with the wrong size is fetched again.
  • Models already broken on disk: when gen.load() fails on truncated weights (safetensors header error, cut-off torch archive), the job error is a short message instead of the traceback: "The model files for X are incomplete or corrupted (path), most likely from an interrupted download. Open the Models page, remove this node's weights, then install them again." The full traceback still goes to the log.

Testing

  • Python: 587 OK. New DownloadFileStreamedTests (early end of stream, resume, wrong-size file, right-size file kept) and test_weight_integrity.py
  • JS tests pass, lint clean, no new tsc errors (Python-only change)
  • Not tested: a real interrupted Hugging Face download and a real TripoSG load

http.client returns b"" when a server closes the stream early instead of
raising, so _download_file_streamed renamed a short .part file into place
and the model looked installed while its weights could never load (#389).

Downloads now check the received size against the size Hugging Face lists
for the file (one list_repo_tree call per repo) or the response length, and
resume from the .part file when it falls short. An existing file whose size
does not match is fetched again instead of being skipped.

For models already broken on disk, a load that fails on a truncated
safetensors header or torch archive now reports a short message pointing to
the Models page instead of the raw traceback.
@lightningpixel
lightningpixel merged commit 3f32302 into dev Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant