Repository navigation
fix(build): stop waiting forever on a hung Docker credential helper (DEP-6855) - #712
Conversation
…DEP-6855) A credential helper that never answers (e.g. docker-credential-desktop with Docker Desktop stopped) held the auth provider's lock, so the builder's auth call timed out and failed the build, and every other registry waited too. Give each helper 50s, wait outside the provider lock, and continue without credentials for that host, logging which helper and host timed out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lookup skipped while its helper was hung or busy stayed empty for the rest of the build, and a timed-out cache refresh dropped credentials that were still valid. Retry skipped lookups after the helper answers, and fall back to the previous answer when a refresh times out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(DEP-6855) The inner provider now only reads the answer the pre-lock wait found, so a helper that recovers in between cannot start a lookup while the lock is held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…ngs (DEP-6855) A helper error was cached for about five minutes; ask the helper again on the next call, as before. Mark a helper hung only while a call to it is still running, so a finished call cannot leave the flag set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45d6e792de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Thanks for the follow-up commit. Both points from my earlier review look fixed in 45d6e79, and this pass found no new bugs. Because this is new concurrent code on the registry auth path, a human look would still be worthwhile.
What was reviewed:
- Error caching:
expired()now treats an errored entry as stale, so the next call asks the helper again. Errored entries are also skipped whenpreviousis chosen. - Hung flag: it is now set only while the helper's slot is held (
len(helperSlots[helper]) > 0), and theloadthat frees the slot clears it. - auth.go:
getwaits before the inner provider takes its lock, and the inner provider only readscached(host).SetLoggeris passed on from both wrapper providers.
Extended reasoning...
The change adds a time-limited, per-host cached Docker credential helper lookup in pkg/registry/credential_helper.go and routes buildkit's auth provider in pkg/registry/auth.go through it. That puts it on the registry credential path. The code is concurrency-heavy, with goroutines, per-helper slot channels, deadlines and shared maps behind a mutex, which is why a human look is still worthwhile. The latest commit addresses both of the earlier inline findings (errors were cached and the hung flag could stay stuck). There are no CODEOWNERS files, and the only other reviewer, a bot, approved.
…t (DEP-6855) An occupied helper slot could belong to a newer lookup still within its own time limit. Mark the helper hung only when this lookup's own call is running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a3ce97758
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The lookup and the cached read now normalize Docker Hub the same way, so they agree whichever host name buildkit passes. Adds a test that Docker Hub credentials from the config are still used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8a0e837. Configure here.
| return entry.previous.authConfig, entry.previous.err | ||
| default: | ||
| return types.AuthConfig{}, nil | ||
| } |
There was a problem hiding this comment.
Timeout fallback lost to late errors
Medium Severity
get can time out and choose empty credentials or the last good previous result, but Credentials discards that answer and re-reads through cached. If GetAuthConfig finishes with an error in between, cached prefers the late answered error over the timeout fallback, so the auth RPC fails instead of continuing without credentials.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 8a0e837. Configure here.


Summary
A stuck credential helper now costs one host 50s and one log line, not the whole build. 🔑⏱️
What was happening
When a Docker credential helper hung (e.g.
docker-credential-desktopwith Docker Desktop stopped), the CLI waited forever while holding buildkit's auth lock. The builder gave up after 60s and failed the build, and every other registry waited behind it.What happens now
Each helper gets 50s, which fits inside the builder's 60s limit with room for its session wait and the round trip. The wait happens outside the lock, so other hosts aren't blocked. On timeout, the CLI logs
[auth] docker-credential-<name> did not respond within 50s for <host>; continuing without credentials for this hostand carries on without credentials: public images pull, private ones get the registry's 401.Anything else
Fixes DEP-6855. Calls to one helper still run one at a time (docker/cli#1862). The
imagetoolspaths that read Docker auth directly are unchanged.🤖 Generated with Claude Code
Catch up on this pull request in Depot
Note
Medium Risk
Touches registry authentication during builds; failures degrade to unauthenticated pulls rather than indefinite hangs, but mis-timeouts could skip credentials for private registries.
Overview
Adds a
credentialLookuplayer that runs Docker credential helpers with a 50s deadline (inside the builder’s 60s auth window), caches successful results, and serializes calls per helper while allowing different helpers in parallel.dockerAuthServernow resolves credentials viaget()before delegating to BuildKit’s locked provider, so a stuck helper blocks only that host. On timeout it logs once (build progress or stderr) and continues with empty credentials rather than hanging the whole build; hung helpers are flagged so other hosts sharing that helper aren’t re-probed immediately.AuthProviderandDepotAuthProviderforwardSetLoggerto surface timeout messages in build output. New non-Windows tests cover hung/slow/flaky helpers, cross-host concurrency, error non-caching, and Docker Hub host key mapping.Reviewed by Cursor Bugbot for commit 8a0e837. Bugbot is set up for automated code reviews on this repo. Configure here.