Repository navigation
Preserve original query failures and fence hydration before handler execution #143
Description
Current delivery status
Published Python 2.5.15, tag 35aad4ab344cf13e01de3e5f4704380eac1532a6, delivers the caller and managed-worker query-authority corrections. Normal installation and three real Server 2.8.0 HTTP originals pass. Installed qualification. Exact stable documentation is deployed.
Sample #223 consumer and published-image acceptance now passes on both amd64 and arm64, with all candidate and fresh-main SDK checks passing. The promised downstream gate is complete. Original Cloud observations and frozen tuples are unchanged, with no retrospective cause claim.
Original finding and pre-implementation boundaries
Normally installed Python SDK 2.5.10 (source caf006be741ad928f4f5e5d7283dce37c2e99f02, wheel ea2535c2ddb96a2891b4f9dbafd8504a9d91205ae6852f3ae9a8bca3d8ddd5c7) against unchanged published Server 2.6.1, using default protocol 1.20, exposes two query boundaries in an independently instrumented local reference.
The original query was claimed 108.890840 seconds after admission. Its caller received HTTP 504 / query_worker_execution_timeout at the original 180-second boundary, while the worker was still loading external payloads. The worker began execution 314.860777 seconds after the poll response, ran the query handler, then received HTTP 409 / query_task_timed_out. Execution began after both the canonical timeout and the original lease expiry. The handler itself took less than one millisecond.
The poll contained 21 external-reference occurrences for three distinct references. Eleven completed GETs transferred 123,033,328 bytes before execution, with 299.465390 seconds of summed transfer time. Repeated reads are measured; cache eviction events were not observed, so this is not a cache-defect attribution.
The default Client transport automatically submitted a second, distinct query after the first typed 504, then a third after the second query's 504 / query_task_not_claimed. These admissions cannot replace the first original failure. No logical query success was observed.
The matching published SDK source explains the boundaries:
Client._request()hydrates runtime-managed references before returning the query task to Worker. The workflow hydration ownership observer applies to workflow polls/history, not query polls.- Query execution metrics and interceptors start after hydration. The query path does not check the original lease expiry before executing the handler.
TransportRetryPolicyretries generic HTTP 5xx before_request()converts the typed response into the SDK error. A query POST therefore admits another query on the observed canonical 504.- The captured query poll exposes lease expiry, but no original caller deadline. Lease expiry and caller timeout are distinct bounds; workflow-task repair observations do not establish query authority.
Before implementation, settle the narrow SDK contract: preserve the first canonical query failure; include loading in the task's existing authority/budget and refuse execution after an expired lease; identify the actual Server boundary needed to observe an earlier canonical query timeout. Keep default 1.20 and explicit minimum 1.19 qualification separate. Do not extend deadlines, invent worker authority, or replace the failed original with a retry. Investigate repeated-reference loading with integrity and memory limits preserved.
Qualification must use normally installed candidate and published controls, retain original task identities and typed errors, prove expired-task handler suppression, and cover ordinary successful queries plus the existing transport retry controls. Any Server contract change is separately coordinated with its owner. No SDK implementation has started for this issue.
This is a fresh reference finding. It does not establish the cause of earlier incompletely instrumented stalls or qualify other SDKs, worker takeover, or a new backend capability.
Activity
rmcdaniel commentedon Oct 13, 2026
Read-only inspection of the exact published Server 2.6.1 task serializer confirms it supplies
lease_ownerandlease_expires_at, but omits the original query admission time/caller deadline. The broker stores admission time internally. Advertised timeout durations alone cannot reconstruct the original remaining caller budget after queueing.The SDK can enforce the exposed original lease bound without claiming that it has observed an earlier canonical timeout. Original-deadline/status authority needs a separate Server contract decision and qualification with its owner. This finding is recorded before implementation; no deadline or published cohort has changed.
rmcdaniel commentedon Oct 13, 2026
The caller retry correction is delivered in Python 2.5.12, source
db0c029b930e5f31eff461eb642cca0c7a8c0257. A canonical query timeout now raises the original typedQueryFailedwithout automatically admitting another query. The existing three timeout reasons, control-plane v2 routes and namespaced URLs are covered; unrelated transport failures retain their previous retry behavior.Complete source CI, independent stable CI and publication/registry checks pass. Fresh normal PyPI installation passes 239 client/retry tests; all 33 package files match the release source, installed wheel and registry sdist. Wheel SHA256
2fbae2030c9eae61eb77125f4c2aa05f34ca121b0ee725eccc35a8cbef3ec41d; sdist66a2ac005084814a7072c834e4fb07dfa34ba8ea3a6071f94ca74534ac34d5b9.Actual normally installed published 2.5.12 HTTP controls pass under default 1.20 and explicit minimum 1.19 against unchanged immutable Server 2.6.1 / MySQL / Redis. Each independently admitted synthetic workflow preserves its first original 504 /
query_worker_execution_timeoutbody, sends one slow-query POST and returns that failure after 3.10–3.18 seconds. Ordinary fast queries return 7. The separate published 2.5.11 controls send three POSTs and expose their final timeout after 9.66–9.82 seconds. The control uses its own 3/8-second query/lease limits; existing failed references and original deadlines are unchanged.All control stacks/drivers are retired. Worker and workflow code remain byte-identical to 2.5.10/2.5.11; the Client change is documentation, and the transport policy carries the fix. Pending main Nexus producer work is excluded.
#143 remains open: this release fixes the caller retry boundary. Query hydration, handler authority and the earlier canonical deadline/status boundary remain to be coordinated with Server #433. No original Source600 cause, full loading-envelope fix, new backend capability or frozen-cohort adoption is claimed.
rmcdaniel commentedon Oct 14, 2026
The remaining worker correction will use the agreed Server #433 contract. Its PHP implementation is still in source qualification. The caller retry correction in 2.5.12 and deterministic start clock in 2.5.13 remain separate qualified behavior.
Before implementation, the SDK boundary is:
query_task_authority=trueand the selected protocol is 1.20, observe the original task/attempt/owner without renewal before loading, during loading, after loading, and after replay immediately before invoking the query handler. Validate immutable timing and identity. A missing, malformed, denied or unavailable authority response suppresses execution.require_query_task_authorityworker option. When selected, discovery must prove the 1.20 authority capability before registration; an old Server or explicit 1.19 is refused. Existing workers can retain the documented compatibility path.Qualification will include old published Server controls, the exact qualified new PHP source and later its published image, default 1.20 and explicit 1.19, slow external loading, replay consuming the deadline, replacement/stale claims, ordinary successful handlers and lost-acknowledgment reconciliation. No native capability, historical Source cause or frozen-cohort promotion follows from this plan.
rmcdaniel commentedon Oct 15, 2026
Version correction: the final registry/tag check found the already published documentation-only 2.5.14 (
67be8f0f05301667521e9ca4c59e3223163e7272, published October 14). It remains immutable and unchanged. The query candidate was never tagged or published to PyPI.The query implementation merged through #152 is being prepared as 2.5.15. The follow-up also restores the published 2.5.14 backfill docstring/reference page/changelog, preserving its start-inclusive, end-exclusive contract. Existing query executable logic remains unchanged; only the restored Client docstring changes that source file, plus version/docs metadata.
Candidate
f493e6f2b4f1600751b3b59839ab0819b0661faa/ tree4733ccc04ddc9e0720c9c15ef9962216a765c2e0has both the query merge and published 2.5.14 as parents. It requires its own exact gates and normal registry-installed acceptance. The earlier private 2.5.14 wheel controls remain implementation evidence, not a public version identity or release claim.rmcdaniel commentedon Oct 15, 2026
The query runtime source and compatible release remain separately gated. Public documentation follow-through also needs an exact published tag: the existing portal deploys main, which includes source-only Nexus APIs. I will add a separate documentation-only workflow change using the existing PHP portal pattern: explicitly select a published stable tag, verify its manifest and release, build that exact source, and prevent prerelease main from replacing stable public docs. No SDK runtime or release identity changes are included.
rmcdaniel commentedon Oct 15, 2026
The compatible package 2.5.15 is now published from immutable tag/merge 35aad4a, tree 4733ccc0. Protected publication 38669469407 passes. PyPI wheel SHA-256 b905d7e440d5d3708c4570ad8c8986d2ff83149bee6ba4bce209992ad3e8a7dc; sdist 7b3b1c0cebab2e09faa32c7b515cef8932f1aab546bfce54a187778eb88a9aec. A fresh normal unpinned index install resolves 2.5.15, all 33 Python modules plus py.typed match the qualified tag, and 131 installed query/storage/replay controls pass. Actual published-package HTTP acceptance is now running separately; final delivery and downstream adoption remain pending. The first install harness used a non-executable temporary filesystem and stopped before tests; its corrected clean harness passed. Existing 2.5.14 is untouched.
rmcdaniel commentedon Oct 15, 2026
Published-artifact HTTP acceptance now passes on normally installed 2.5.15 and immutable Server 2.8.0 / MySQL 8.4 / shared Redis. Three separate originals retain their four-second query budgets: timely typed int64/null/Unicode result (one GET, one completion); expiry while loading (one GET cancelled, zero handler callbacks/completions); expiry during async handling (handler cancelled and joined, zero completion). Same original task IDs and unchanged workflow histories are verified by the committed opt-in test. The first separate SQLite/array-cache fixture failed with all queries unclaimed and remains unsuccessful; its process-local cache cannot qualify the shared broker path. Its queue shutdown137 also remains unsuccessful. No original deadline changed and no historical Cloud cause is inferred. Selected stable portal154/ffd8be65 is merged; deployment38669900601 passes and live deployment-audit.json binds exactly 2.5.15/35aad4ab. Main/source alignment and Sample consumer pin remain the next delivery gates.
rmcdaniel commentedon Oct 15, 2026
Delivered in normally installed Python 2.5.15, tag
35aad4ab344cf13e01de3e5f4704380eac1532a6. The installed caller/managed-worker authority qualification passes.The final Sample consumer/image gate also passes, including all candidate/fresh-main SDK profiles and both published image architectures. Closing as delivered. Original Cloud failures and their historical attribution remain unchanged; this does not claim native or Cloud qualification.