Repository navigation
Wait for other jobs only before submitting to the runner - #4390
Merged
Merged
Conversation
`_prepare_startup_context` ran on every PROVISIONING and PULLING tick and returned early whenever the runner submission had to wait for other jobs: replica provisioning, the router IP, or referenced group IPs. The early return also skipped the status handlers' own work: copying the instance provisioning data into the job, failing the job when its instance is terminated, the provisioning timeout, and shim polling. With replica IP refs on newly provisioned instances, jobs waited for IPs that only the skipped copy would set, so the run stayed in provisioning forever. * Replace `_prepare_startup_context` with `_prepare_runner_submission`, called right before the runner submission once the runner is up. * Load volumes and interpolate secrets in the shim branch, which needs the interpolated `registry_auth`. The shim no longer waits for other jobs, so the image pull starts earlier. * Terminate workers whose router failed or did not get an IP in time in `_terminate_if_router_unavailable`, before both statuses as before. * Wait for every job in the replica to have a hostname, not only provisioning data, since shim submission no longer ensures it. * Drop the unused `secrets` parameter of `_submit_job_to_runner`. Fixes: #4331 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_prepare_startup_contextran on every PROVISIONING and PULLING tick and returned early whenever the runner submission had to wait for other jobs: replica provisioning, the router IP, or referenced group IPs. The early return also skipped the status handlers' own work: copying the instance provisioning data into the job, failing the job when its instance is terminated, the provisioning timeout, and shim polling. With replica IP refs on newly provisioned instances, jobs waited for IPs that only the skipped copy would set, so the run stayed in provisioning forever._prepare_startup_contextwith_prepare_runner_submission, called right before the runner submission once the runner is up.registry_auth. The shim no longer waits for other jobs, so the image pull starts earlier._terminate_if_router_unavailable, before both statuses as before.secretsparameter of_submit_job_to_runner.Fixes: #4331