Skip to content

fix(build): stop waiting forever on a hung Docker credential helper (DEP-6855) - #712

Merged
121watts merged 7 commits into
mainfrom
watts/dep-6855-depot-cli-never-answers-the-builders
Nov 20, 2026
Merged

121watts merged 7 commits into
mainfrom
watts/dep-6855-depot-cli-never-answers-the-builders

Conversation

@121watts

@121watts 121watts commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A stuck credential helper now costs one host 50s and one log line, not the whole build. 🔑⏱️

What was happening

When a Docker credential helper hung (e.g. docker-credential-desktop with Docker Desktop stopped), the CLI waited forever while holding buildkit's auth lock. The builder gave up after 60s and failed the build, and every other registry waited behind it.

What happens now

Each helper gets 50s, which fits inside the builder's 60s limit with room for its session wait and the round trip. The wait happens outside the lock, so other hosts aren't blocked. On timeout, the CLI logs [auth] docker-credential-<name> did not respond within 50s for <host>; continuing without credentials for this host and carries on without credentials: public images pull, private ones get the registry's 401.

Anything else

Fixes DEP-6855. Calls to one helper still run one at a time (docker/cli#1862). The imagetools paths that read Docker auth directly are unchanged.

🤖 Generated with Claude Code

Catch up on this pull request in Depot


Note

Medium Risk
Touches registry authentication during builds; failures degrade to unauthenticated pulls rather than indefinite hangs, but mis-timeouts could skip credentials for private registries.

Overview
Adds a credentialLookup layer that runs Docker credential helpers with a 50s deadline (inside the builder’s 60s auth window), caches successful results, and serializes calls per helper while allowing different helpers in parallel.

dockerAuthServer now resolves credentials via get() before delegating to BuildKit’s locked provider, so a stuck helper blocks only that host. On timeout it logs once (build progress or stderr) and continues with empty credentials rather than hanging the whole build; hung helpers are flagged so other hosts sharing that helper aren’t re-probed immediately.

AuthProvider and DepotAuthProvider forward SetLogger to surface timeout messages in build output. New non-Windows tests cover hung/slow/flaky helpers, cross-host concurrency, error non-caching, and Docker Hub host key mapping.

Reviewed by Cursor Bugbot for commit 8a0e837. Bugbot is set up for automated code reviews on this repo. Configure here.

…DEP-6855)

A credential helper that never answers (e.g. docker-credential-desktop with
Docker Desktop stopped) held the auth provider's lock, so the builder's auth
call timed out and failed the build, and every other registry waited too.

Give each helper 50s, wait outside the provider lock, and continue without
credentials for that host, logging which helper and host timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 20, 2026

Copy link
Copy Markdown

DEP-6855

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/registry/credential_helper.go Resolved
A lookup skipped while its helper was hung or busy stayed empty for the
rest of the build, and a timed-out cache refresh dropped credentials that
were still valid. Retry skipped lookups after the helper answers, and fall
back to the previous answer when a refresh times out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/registry/credential_helper.go Resolved
…(DEP-6855)

The inner provider now only reads the answer the pre-lock wait found, so a
helper that recovers in between cannot start a lookup while the lock is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@121watts
121watts marked this pull request as ready for review November 20, 2026 13:34
cursor[bot]
cursor Bot approved these changes Nov 20, 2026 •
@121watts

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T14:13:15.865522Z aea6ffb Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread pkg/registry/credential_helper.go Resolved
Comment thread pkg/registry/credential_helper.go Resolved
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 50835ee3e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ngs (DEP-6855)

A helper error was cached for about five minutes; ask the helper again on
the next call, as before. Mark a helper hung only while a call to it is
still running, so a finished call cannot leave the flag set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@121watts

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45d6e792de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/registry/credential_helper.go Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up commit. Both points from my earlier review look fixed in 45d6e79, and this pass found no new bugs. Because this is new concurrent code on the registry auth path, a human look would still be worthwhile.

What was reviewed:

  • Error caching: expired() now treats an errored entry as stale, so the next call asks the helper again. Errored entries are also skipped when previous is chosen.
  • Hung flag: it is now set only while the helper's slot is held (len(helperSlots[helper]) > 0), and the load that frees the slot clears it.
  • auth.go: get waits before the inner provider takes its lock, and the inner provider only reads cached(host). SetLogger is passed on from both wrapper providers.
Extended reasoning...

The change adds a time-limited, per-host cached Docker credential helper lookup in pkg/registry/credential_helper.go and routes buildkit's auth provider in pkg/registry/auth.go through it. That puts it on the registry credential path. The code is concurrency-heavy, with goroutines, per-helper slot channels, deadlines and shared maps behind a mutex, which is why a human look is still worthwhile. The latest commit addresses both of the earlier inline findings (errors were cached and the hung flag could stay stuck). There are no CODEOWNERS files, and the only other reviewer, a bot, approved.

…t (DEP-6855)

An occupied helper slot could belong to a newer lookup still within its own
time limit. Mark the helper hung only when this lookup's own call is running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@121watts

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a3ce97758

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/registry/credential_helper.go Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

The lookup and the cached read now normalize Docker Hub the same way, so
they agree whichever host name buildkit passes. Adds a test that Docker Hub
credentials from the config are still used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@121watts

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: aea6ffbc3e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@121watts
121watts enabled auto-merge November 20, 2026 17:14
@121watts
121watts merged commit a9279aa into main Nov 20, 2026
13 checks passed
@121watts
121watts deleted the watts/dep-6855-depot-cli-never-answers-the-builders branch November 20, 2026 17:15

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8a0e837. Configure here.

return entry.previous.authConfig, entry.previous.err
default:
return types.AuthConfig{}, nil
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeout fallback lost to late errors

Medium Severity

get can time out and choose empty credentials or the last good previous result, but Credentials discards that answer and re-reads through cached. If GetAuthConfig finishes with an error in between, cached prefers the late answered error over the timeout fallback, so the auth RPC fails instead of continuing without credentials.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8a0e837. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant