Repository navigation
fix(models): never treat a truncated weight download as complete - #465
Merged
Merged
Conversation
http.client returns b"" when a server closes the stream early instead of raising, so _download_file_streamed renamed a short .part file into place and the model looked installed while its weights could never load (#389). Downloads now check the received size against the size Hugging Face lists for the file (one list_repo_tree call per repo) or the response length, and resume from the .part file when it falls short. An existing file whose size does not match is fetched again instead of being skipped. For models already broken on disk, a load that fails on a truncated safetensors header or torch archive now reports a short message pointing to the Models page instead of the raw traceback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #389
TripoSG failed to load with
SafetensorError: Error while deserializing header: incomplete metadata, file not fully covered: a weight file on disk was truncated, yet Modly reported the model as downloaded, and every Generate showed a raw traceback.Root cause
_download_file_streamed(api/routers/model.py) treated an early end of stream as success. When the server closed the connection early,http.clientreturnedb""instead of raising, and the short.partfile was renamed into place. After that, any non-empty final file was skipped, so downloading again never repaired it.Changes
HfApi.list_repo_tree(one call, sizes included) instead oflist_repo_files.IncompleteDownload. The existing retry loop then resumes from the.partwith a Range request.gen.load()fails on truncated weights (safetensors header error, cut-off torch archive), the job error is a short message instead of the traceback: "The model files for X are incomplete or corrupted (path), most likely from an interrupted download. Open the Models page, remove this node's weights, then install them again." The full traceback still goes to the log.Testing
DownloadFileStreamedTests(early end of stream, resume, wrong-size file, right-size file kept) andtest_weight_integrity.pytscerrors (Python-only change)