Skip to content

MAAP credential broker: jittered backoff within a 240 s budget, one client, stop on 401 - #62

Merged
SmithB merged 1 commit into
mainfrom
maap_broker_backoff
Oct 7, 2026
Merged

SmithB merged 1 commit into
mainfrom
maap_broker_backoff

Conversation

@SmithB

@SmithB SmithB commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Follows #61. When several hundred DPS jobs start together the MAAP API stops answering, and five tries 10 s apart, each building a new MAAP() (itself an API call), still lose jobs.

_s3fs_from_maap now:

  • pauses 10, 20, 40, 60, 60 s between tries (up to 6), each × uniform(0.5, 1.5) so jobs that failed together do not retry together;
  • takes no pause that would end past 240 s from the first try;
  • keeps one MAAP client across tries (built again only if building it failed);
  • stops at the first HTTP 401: MAAP_PGT was rejected, and retrying the same token cannot help.

There is no per-try timeout, because this may run off the main thread, where SIGALRM is not available. The schedule matches ATL1415's workspace_credentials.py (plan_pack_tiles.sh K2–K3). MAAP_BROKER_ATTEMPTS / MAAP_BROKER_PAUSE_S are replaced by MAAP_BROKER_PAUSES_S, MAAP_BROKER_JITTER, MAAP_BROKER_BUDGET_S; nothing else referenced them.

Tests: tests/test_maap_broker.py 16 passed (5 new: jitter, budget, 401, client reuse, rebuild after a failed build); full suite 323 passed, 3 skipped.

🤖 Generated with Claude Code

…t, one MAAP client across tries, stop at once on HTTP 401

Was 5 tries 10 s apart, with a new MAAP() (itself an API call) for each.
Now pauses 10, 20, 40, 60, 60 s, each x uniform(0.5, 1.5), with no pause that
would end past 240 s from the first try; the same schedule as ATL1415's
workspace_credentials.py.  A 401 means MAAP_PGT was rejected and a retry
cannot help.  No per-try timeout: this may run off the main thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SmithB
SmithB merged commit 8aad339 into main Oct 7, 2026
2 checks passed
@SmithB
SmithB deleted the maap_broker_backoff branch October 7, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant