Skip to content

Keep embedded database-queue recovery bounded during concurrent fan-out #809

Closed

Description

@rmcdaniel

Problem

A concurrent embedded database-queue fan-out can accumulate repeated recovery deliveries while the original activities remain pending. Separate queue-only controls encounter SQLite lock errors before activity callbacks and during transport acknowledgment.

This was found while qualifying #773, then reproduced with a normal Composer installation of Workflow 2.5.23 (31cc009e24f2282b59ca3cff00d389f1cafe0171), Laravel 13.30.1, PHP 8.5.11 and SQLite 3.53.4. The published package has no draft tracing instrumentation. The claim and SQLite transaction implementation bytes match the released tag.

Observations

The workload creates 64 ordinary activities through all(), using four independent queue:work processes, a 2-CPU/2-GiB envelope, SQLite DELETE journal and its ordinary 5,000-ms busy timeout. Original execution/run deadlines are 120/100 seconds, with a 90-second observer. Every activity returns the same typed bigint/null/Unicode structure plus its member number.

Separate published original Queue/matching mode Observed outcome
01m51mjv5313yn8v1r4fppfp48 Queue-only, 45-second worker lifecycle 61 completed activities, three pending, no failed-job row; unfinished.
01m51mnk575wzshpq1tfgwszyz Queue-only, added bounded delivery diagnostics 61 completed, two pending, one running; two failed-job rows and three reserved queue entries.
01m51n18cmfj9w33gwhrea9ya9 Normal matching enabled, 45-second worker lifecycle 35 callbacks, 191 repairs and 220 ready queue entries; unfinished, with zero failed-job rows.
01m51n4z9ez6q128eympc5mcw6 Fresh 80-second worker-lifecycle diagnostic 25 callbacks, 264 repairs, 303 queued entries for 63 distinct logical tasks, up to 11 transport copies per task. This control overran the original observer and remains failed, not accepted qualification.

The diagnosed queue-only original records two distinct boundaries:

  • SqliteTransaction::forTask()'s first workflow_tasks.id = id update, called by ActivityTaskClaimer before callback execution.
  • SqliteAcknowledgingDatabaseQueue::deleteClaimedJob()'s fenced transport deletion.

Both report SQLite error 5. Bounded exception observation records a preceding root rollback with framework transaction level zero and PDO outside a transaction. This observation does not prove commit/acknowledgment authority for other failures. The first diagnostic did not retain enough detail to explain its missing delivery.

All originals preserve their recorded history prefixes and original execution/run deadlines. None establishes completed fan-out acceptance. The claim failures, acknowledgment failure, repair amplification and worker lifecycle must be distinguished before choosing a correction.

Acceptance

  • Establish the repair/transport relationship with a deterministic source counterfactual, including tasks that already have queued deliveries and genuinely missing deliveries.
  • Bound recovery work under a healthy backlog while preserving supported missing-job and expired-lease recovery, namespace/queue isolation and application-owned connectors.
  • Keep the original task identities, complete policies, history and deadlines. Execute each activity callback once and complete the same original workflow.
  • Exercise claim and acknowledgment pressure separately. No blind re-enqueue, swallowed failed() or retry of uncertain commit/rollback acknowledgment.
  • Make the continuous-worker observer honor its original deadline. Preserve the failed lifecycle originals rather than relabeling them.
  • Run source gates, qualify a compatible published patch and normally installed consumers if a runtime correction is required.

General owns this package follow-through. #810/#811 delivered the bounded queue-recovery correction in published 2.5.24, with all 16 supported published upgrade profiles passing. Its first normal-installed original completes all 64 activities but still fails clean acceptance with one failed pre-claim delivery and one repair: outcome.

#812 adds the separately counterfactually tested SQLite write-admission correction. Exact source fresh-main 38880772869 and compatible #813 candidate 38881012149 pass. Published 2.5.25 is immutable 1ff2a82baa6586c1048656898d91805b29bee20f / tree 09013fdc368719e21e3017855be62a6945f8364b; registry/platform and all 16 supported published upgrade profiles pass in 38898397689.

The first normally installed 2.5.25 original passes the unchanged 64-activity/four-continuous-worker profile: exact output, history prefix and original deadlines, callbacks once, zero failed deliveries/repairs/remaining jobs. Outcome. A bounded successful original is not a contention-free or capacity claim.

Sample #230 is delivered at 5ac85799, with both root/microservice locks on 2.5.25 and both published native images qualified. The normal embedded Waterline 2.11.1 consumer also passes its actual read-only Laravel HTTP-kernel views, including wrong-namespace refusal and the hidden queue-delivery hint. These outcomes do not replace the earlier failed originals.

#814 / #815 deliver the separate fenced acknowledgment correction in published 2.5.26, exact ff61e0f04495de10d83f300492b1b866f1fc694b. All 17 published registry/upgrade checks and 44 normal-installed acknowledgment/embedded controls pass. Sample #231 is its separately pinned downstream image follow-through.

Source #816 is delivered at 1ec8b5ed. It retains an in-memory result across proven pre-publication root rollbacks within the original claim/deadlines, with fresh expiry/replacement checks and no callback reexecution. Compatible #817 publishes 2.5.27, immutable 8ba5bec6f5fb72c6f159f4b82695f19b06017e63 / tree 24908d95. All 17 published profiles pass, and normal-installed pressure/expiry/replacement qualification passes 78 controls / 895 assertions. Delivery. Sample #232 delivers both locked consumers and published native images.

The normally installed PHP 8.5 continuous-worker original completes all 64 activities exactly within the original 90 seconds, but has three repairs and remains unsuccessful for the strict zero-repair metric. #818 is merged at a51bdf64 / reviewed tree e984e9d6. Its exact b81 normally installed PHP 8.3 original verifies the shared 2 CPU / 2 GiB / no-swap scope, then completes 64 first-attempt callbacks with zero repairs/failed jobs/delivery exceptions/remaining entries and unchanged typed results/history/deadlines in 41.51 seconds. Receipt. The independent exact-main published consumer and full fresh-main qualification pass every applicable gate. All six acceptance criteria are audited in the final disposition. Earlier debugger-enabled and lifecycle failures stay failed.

The stronger tracing fan-out gate remains selected in draft #788. Its f141 hosted traced/disabled originals time out at 48/49 completed activities and remain failed. Fresh integration 8b18 now independently passes all 145 selected tracing controls / 10,951 assertions, including both large fan-outs with original observers. Its aggregate still fails the separate released OTel API encoded-baggage case. No full tracing, performance or general contention-free claim follows. All earlier failed originals retain their outcomes.

Activity

  1. added
    bugSomething isn't working
    kind:defectA public product behavior is incorrect
    priority:P1High-priority product or release risk
    intake:approvedCurrent issue title and body revision is approved for authority intake
    status:in-progressApproved work is actively being implemented or validated
    on Oct 16, 2026
  2. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    The deterministic source counterfactual now has three failures / six passing controls (9 tests, 87 assertions). Existing code republishes an unreserved queued workflow, activity or timer task solely because four seconds elapsed. Genuinely missing and reserved-delivery controls retain their current recovery behavior.

    Proposed narrow correction before implementation:

    • Record an optional hidden database-queue delivery reference with the existing successful task-dispatch update: original task/run/namespace, resolved connection/queue/table and returned job primary key.
    • Add a small task-ID marker through Laravel's existing payload hook for our three job classes only. Probe that known row by primary key, queue and expected job/task identity. Do not scan the queue or deserialize queued objects.
    • Before repair locks, an exact unreserved delivery can suppress this redundant repair pass. Its presence grants no claim or completion authority. If the row disappears immediately afterward, the next pass rechecks it. Reserved, missing, mismatched, malformed or unavailable references retain the existing recovery path.
    • Keep custom application queue managers/connectors and poll mode unchanged. Old schemas/jobs without usable metadata fall back to current behavior. A nullable internal task column is required to avoid a per-task full queue scan; it is hidden from task JSON and excluded from business history/payload/equality.
    • Keep existing claim/lease/deadline checks, task-repair timing and failure policy. Claim/acknowledgment pressure and full concurrent acceptance still need qualification after this correction.

    This is unpublished Workflow package scope. Native catalogs/shared fixtures, service wire, current/frozen tuples and Server dependencies do not adopt the new optional column. No version, release or broader capability claim follows from the source proposal.

  3. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    The corrected first component passes 9 tests / 120 assertions. The unchanged source fails the three queued workflow/activity/timer controls; the six missing/reserved controls pass both before and after the change.

    The implementation now uses Laravel's existing JobQueued event and payload UUID. It adds no queue-payload field, replaces no payload hooks and deserializes no queued objects. This supersedes the proposed marker hook. Laravel's after-commit dispatch returns no job ID; the native event supplies it. Older Laravel event shapes use a single primary-key read to obtain the existing UUID. The optional hidden reference remains co-recorded with the existing task-dispatch update.

    The exact unreserved row is inspected before run/task repair locks. Reserved, absent, mismatched and legacy references still need negative/rollback/ownership controls. Candidate code, new nullable column, continuous-worker qualification, full source gates and publication remain unfinished. No schema/native catalog, service wire or dependency adoption is implied.

  4. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Source correction is pushed in draft #810 at 3fbbb7ae525b263823f5cc0e0fd470b80d74de07, tree e178f8b856353d160ebca78b3cf08c66a944065b. The original three queued-delivery counterfactuals remain in cd816243; current focused controls pass 34 / 334, and the combined SQLite transaction/reservation/acknowledgment/recovery selection passes 98 / 921. ECS/PHPStan and ten build-routing controls pass.

    The reference uses Laravel's original payload UUID and native enqueue identity. The observer adds no queue-payload field, replaces no hooks and deserializes no objects. A byte-exact unreserved row observation can omit redundant repair before run/task locks; it grants no claim/completion authority. Missing/reserved/legacy and disappearing-row recovery remains tested. The optional hidden column is unpublished and has no native/shared/service/Server adoption.

    Three preceding source originals complete all64callbacks with original deadlines/history; the latter two directly verify64first attempts/195events. Continuous original 01m51q3vzkg8k1q5rwszy42hk5 has exact outputs and no remaining jobs. Each still records one pre-claim failure and one automatic repair, so strict clean acceptance remainsFAIL. No original is rescued or relabeled, and no full concurrency/performance claim follows.

    Original0bf build38863479087 fails before runtime scenarios: I incorrectly put Testbench tests in the plain PHPUnit selector list. The same local selector rejects that configuration. Current source keeps the selector rule, adds a dedicated mandatory embedded step and passes selection. Original failures remain unchanged. Its Sonar/app gate passes0.0%duplication with12 reviewed findings; current source addresses them, with fresh analysis pending and no tuning/threshold change.

    Next: inspect exact-head CI/Sonar, then qualify the combined source and real network-database behavior before publication/normal-installed acceptance. #809 remains open; no release is claimed.

  5. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Draft #810 now has a fresh passing continuous-worker original on source b638a0f3fd3eb4eab4e71631f03f221b8a55cd4a / tree edfdcc9286ada82a179fd772eb02e33e5fdeea60.

    Original 01m51t06v69qvskqbbhdt0kk8e:64 activities/callbacks,64 first attempts,195 events and66 completed tasks. Exact typed output, original history prefix and execution/run deadlines pass. There are zero repairs, failed deliveries and remaining queue jobs, so both business and original strict clean checks pass. Four actual continuous workers keep45-second lifecycles within the original90-second observer and120/100-second workflow limits. Callbacks ran in two roles; no four-way overlap claim. Resources remain2CPU/2GiB/no swap. Loaded source hashes match the commit; this is an explicit unpublished source control using Laravel13.30.1/PHP8.5.11/SQLite3.53.4, DELETE and5000ms. Earlier failed published and recovered-source originals remain unchanged.

    Review added two necessary regressions: queued work filling the first scan batch cannot starve missing delivery behind it (test-first12153b4b), and reading scan positions must not add partial-model observer calls (test-first4488fbb8). Current combined SQLite checks pass101tests/943assertions. Network components pass13/165 each onMySQL8.0.46/PostgreSQL17.11. Poll selection, per-pass limits and authoritative task/run/lease checks remain intact.

    Sonar caught page-planner complexity19/15. The bounded page read/wrap extraction now has exact Sonar38867754984/app PASS, zero unresolved PR issues and0.0% duplication under the unchanged3% limit. No rule/profile/threshold change.

    Full source38867753704 and ordinary38867754849 are still running. Quality/corpus/MariaDB/five upgrade profiles pass so far; remaining database shards/coverage and later compatible publication/installed acceptance are pending. This is not release or complete #809 delivery.

  6. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Final writer-routing review is implemented at 268f333feb09901812afe410ede264ca126cd880 / tree ad9bc6690be9a8d675d673812a685227e3b8c069.

    Controlled read/write PDO fixtures prove both routing errors on the prior code: a lagging empty reader causes redundant repair, while a stale retained row hides genuinely missing delivery (two failures, one unchanged direct-writer control passes). The writer probe passes all three. A separate frozen storage snapshot proves that the canonical task hint also needs a writer read: the old watchdog snapshot creates another delivery; the corrected snapshot preserves the current queued one. These are controlled PDO states, not a claim about a deployed replication system.

    The final change explicitly reads the task hint and native queue row through their writer connections. Existing locks, task/run/lease checks and per-pass limits remain. Current combined SQLite regression selection passes 106tests/978assertions, plus ECS/PHPStan. All short-lived local test containers, copied snapshot inputs, mountpoints and cache were verified absent08:21:19.

    Preceding d8b7b4bf full38870229180, ordinary38870054555 and Sonar38870054423 all pass. Those remain their own exact-tree results. The two writer-routing flags now receive current normalPR38873135265 and Sonar38873135435, followed by automatic full MAIN qualification after merge. Publication still requires that main result and the independent compatible build. This avoids repeating the same full matrix both before and after a small, directly tested routing correction.

    Compatible draft #811 is updated to aa20a06cb43eda316b5abaf6371d27ec3bd3ec00 / tree 845c8a458836d8ad2a4b2fcb3788511990aad39a, based only on published2.5.23. Its changed runtime files match current source. Independent full38873322432 is running. No tag, publication, installed acceptance, Server/frozen adoption or complete #809 claim.

  7. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Published 2.5.24 is available at immutable commit 62a214a7a28c11ac7b94e1f92a59c12eca5a4899, tree 845c8a458836d8ad2a4b2fcb3788511990aad39a. Source #810 fresh-main 38873974454 and compatible #811 full candidate 38873322432 pass. Published verification 38876394511 now passes the registry/platform contract and all 16 supported Laravel/PHP upgrade profiles. The release notes disclose the remaining stress-control failure below.

    The first independent normal Composer installation preserves Laravel 13.30.1 / Testbench 11.2.0 and loads the released package without an autoload override. Original 01m51yhbx6grjj55fvjwscsnsv, four continuous 45-second workers, the original 90-second observer and 120/100-second limits: all 64 typed activities complete exactly once, 66 tasks and 195 history events complete, original prefix/deadlines remain exact, and no queue rows remain. Business invariants pass. Clean acceptance fails: one failed transport delivery and one repair. This is not a clean installed fan-out pass.

    The captured exception is SQLite error 5 in SqliteTransaction::forTask()'s primary-key admission UPDATE, before ActivityTaskClaimer enters its claim body. The recorded rollback is root level zero with PDO outside a transaction. Callback execution belongs to a different successful delivery. This identifies the failing stage, not a complete contention cause or a correction. The original outcome is preserved, no deadline or expectation was relaxed, and no automatic repeat was made. #809 remains open; Sample adoption is held while this remaining pre-claim boundary is investigated.

  8. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Published Workflow 2.5.25 now has its first normal-installed fan-out PASS.

    • Immutable package 1ff2a82baa6586c1048656898d91805b29bee20f / tree 09013fdc368719e21e3017855be62a6945f8364b, resolved from Packagist with no source autoloader override. Laravel 13.30.1 / Testbench 11.2.0 / PHP 8.5.11; SQLite 3.53.4 DELETE, busy timeout 5000.
    • Fresh original 01m529tn5a31easya130wcpjq7: four real continuous 45-second queue worker processes, unchanged 90-second observer and original execution/run deadlines. All 64 callbacks, 64 activities and 64 attempt-one outcomes completed once; 66 completed tasks and 195 events, exactly one workflow completion.
    • Exact bigint/null/Unicode output, whole original history prefix and deadline bytes preserved. Zero failed jobs, delivery exceptions, repairs or remaining transport jobs. Both business invariants and the unchanged strict clean metric PASS. This bounded original does not claim four-way callback overlap, capacity or absence of contention under arbitrary load.

    Source fresh-main 38880772869 and compatible candidate 38881012149 both PASS. Published verification 38898397689 is running; its registry/platform job has passed. Sample's scoped embedded lock/manifest update and installed Waterline follow-through are underway. The Server image remains pinned to its existing package; SDKs, Cloud and reserved native/frozen cohorts are not implicitly adopted.

    The published 2.5.24 and earlier failed source/stress originals stay failed. This result is a new original, not a retry or relabeling of them. #809 remains open for the remaining supported-package and consumer gates.

  9. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Published-package verification 38898397689 PASS: registry/platform and all 16 supported Laravel/PHP upgrade profiles on the immutable 2.5.25 merge 1ff2a82b.

    The unchanged normal 64-activity profile's first 2.5.25 original is already recorded in 6215208657: exact typed output/prefix/deadlines, callbacks once, zero repairs/failed deliveries.

    Embedded Waterline 2.11.1 (b1526b96) with normally installed Workflow 2.5.25 now passes a separate cold one-activity integration. Fresh run 01m52a2gztv0sgtjp0h2cr355k completes one activity/attempt, zero repairs/failed/remaining jobs. Six real Laravel HTTP-kernel reads cover running/list/detail, pending activity, completed/list/detail and wrong-namespace 404. Every read preserves the complete selected database/queue fingerprint. The new transport hint is actually populated and remains hidden from model/API serialization. Bigint/null/Unicode output is exact. This was an HTTP-kernel integration, not a network/browser or fan-out capacity claim, and required no Waterline package edit/release.

    An earlier supplementary Waterline fixture did not reach completion after its in-process 20-second stop-when-empty drain. It remains incomplete; the fixture did not retain enough final-state diagnostics to establish its cause. The new cold one-activity read profile and the normal four-worker 64-activity pass above are separate originals, not rescues or reinterpretations of it.

    Sample #230, head 9d5b9fe2, updates exactly one Composer package and the embedded artifact field. Artifact graph, PHP/microservice tests, boundary and Sonar pass; direct Sonar PR read has zero unresolved findings. Its image/startup and portable/runtime checks remain live, so downstream publication/installed acceptance is pending. Server's independently pinned package, SDKs, Cloud and reserved frozen/native cohorts stay unchanged.

    All local task containers are gone; package/vendor/raw fixture data is being removed after this outcome readback. No bulky local evidence retention is needed. #809 stays open for Sample's exact-head and published image follow-through.

  10. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Separate normally installed 2.5.25 / Laravel 13.37.0 observation: original 01m52avxyf4hjb04901emymywh completes all 64 activities/callbacks/attempt-one outcomes, 66 tasks and 195 events. Exact output, whole original prefix and original 120/100-second execution/run deadlines are preserved inside the same 90-second observer. Four continuous 45-second workers, 2 CPU / 2 GiB, matching enabled. Zero failed jobs, delivery exceptions or remaining queue jobs; one activity repair. Business invariants PASS; unchanged strict zero-repair clean metric FAIL. No repair amplification or callback duplication was observed.

    The normal registry graph is Workflow 1ff2a82b, Laravel 685f8d6c / 13.37.0, Testbench 11.2.0, PHP 8.5.11. This is a separate original from the earlier Laravel 13.30.1 zero-repair pass, which remains valid for its declared graph. It does not explain the failures on the unpublished tracing draft, whose Testbench version and worker/matching profile differ.

    One separate published queue-only control is now running against this same installed graph with only the matching wake setting disabled. Deadlines, continuous workers, budgets and assertions are unchanged. The result above is not being retried or relabeled. #809 remains open; Sample230's corrected image/startup passes, while its final activity-recovery gate is still live.

  11. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    The independent normally installed 2.5.25 queue-only control is FAIL, original 01m52b0c919ceqtqcdj8jrgsdr.

    Same Workflow 1ff2a82b, Laravel 13.37.0, Testbench 11.2.0, PHP 8.5.11, four continuous 45-second workers and unchanged 90-second observer / 120/100-second deadlines as the preceding normal profile. Only matching wake was disabled. Final state: 62 completed activities/callbacks/attempts, two pending activities; 63 completed tasks/two ready; zero repairs; three remaining jobs and one failed-job row. It does not complete business or clean acceptance.

    The failed row records SQLite error 5 in SqliteTransaction::forTask() before the activity claim. A separate delivery exception records the exact original-reservation DELETE in SqliteAcknowledgingDatabaseQueue, with the observed root transaction level 0 / PDO outside a transaction after rollback. These remain separate failure boundaries. The result reproduces a queue-only failure without draft tracing, but does not establish every lock's cause or permit retry of an uncertain acknowledgment.

    The next narrow source step is the fenced acknowledgment retry budget, with real two-connection counterfactuals and unchanged uncertainty/fence guards. Claim-pressure disposition and full sustained acceptance remain open. No repeated run, deadline increase or assertion relaxation is being used to rescue these originals. All package/database/raw fixture state will be deleted after this concise outcome is read back.

  12. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    The acknowledgment correction is now published as 2.5.26, exact ff61e0f0, with a normal installed Laravel 13.37 consumer passing 44 controls / 718 assertions. Published upgrade and downstream Sample qualification continue on #815. This does not close the remaining outcome-publication boundary.

    Draft #816 contains a focused reproduction and correction for that boundary. On the parent, a real writer forces a 5.166-second admission failure after the callback; the writer releases after the proven root rollback, but the original task remains leased and the delivery fails despite its five-minute claim still being valid. The callback count is one.

    Candidate 960cac22a43e263994e9c4fc43b7d5cd6ccd0d9e, tree 3dba88e78c1896f8939d2bb9a1f9108ba0198202, retains the outcome while retrying only proven admission rollbacks within a monotonic bound captured from the original claim and earliest existing activity deadline. Fresh stored authority still refuses expired or replaced attempts. The callback and an entered publication body do not repeat. Local 66 tests / 843 assertions, ECS and production PHPStan pass; full 38907403356, normal PR 38907592735 and Sonar 38907592833 are independently pending. No compatible publication or installed claim for this correction yet.

    The earlier queue-only and tracing failures remain failed. Full #809 acceptance and full tracing remain unfinished; no blanket SQLite contention-free claim is made.

  13. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Workflow 2.5.27 is published at immutable 8ba5bec6f5fb72c6f159f4b82695f19b06017e63, tree 24908d959ade35d6c11fc1e2e654a5a789857ce4. Exact source fresh-main 38912375676, compatible candidate 38910103083 and published verification 38915513763 all PASS, including all 17 published registry/platform/upgrade jobs.

    A normal Composer installation loads all four changed runtime classes from the installed package, with matching released hashes. PHP 8.5.11 / Laravel 13.37.0 / Testbench 11.2.0 / PHPUnit 13.4.3 passes 78 controls / 895 assertions, including the actual 5,100 ms second-PDO writer after a callback, preserved original history/deadlines, callback once, and stored expiry/replacement refusal. No source runtime autoload override is used.

    The first separate four-worker fan-out is FAILURE, original 01m52hq24mba0y64xdjysyt0zs. All 64 activities and callbacks complete once on attempt one, with zero repairs or failed jobs, but the workflow remains Waiting at 194 events / 65 completed tasks. Delivery exception reporting was not instrumented in this fixture. A single original workflow task is Ready/unclaimed with its actual transport job present. It is not a complete workflow acceptance.

    Stopped read-only SQL locates the boundary precisely: workers entered their ready barrier at 14:26:49.60 UTC; the last activity completion is 14:27:34.938794 and the new parent task is available at 14:27:35.006824. This is at the fixture's agent-selected 45-second worker lifetime. Original workflow deadlines remain 14:28:14.761357 / 14:28:34.761357. The SQLite 3.53.4 DELETE database in this consumer is on the host bind mount, with 5,000 ms busy timeout.

    The fixture should keep its four ordinary workers serving until the already defined 90-second observer completes or expires. The 45-second independent exit is a fixture setting rather than a workflow deadline. I will correct only that prospective consumer lifecycle: same four roles, 2 CPU / 2 GiB, configured 30-second per-job timeout, original 90-second observer and 120/100-second workflow deadlines. Stop workers after observed completion, then perform the same cold read and invariants. The failed original above remains failed; it will not be resumed, repaired or relabeled. This does not explain or fix the separate hosted tracing pre-callback refusals.

    Sample #232 now updates only the two embedded locks and manifest to this published package; candidate qualification is pending. Server's independent pinned package and native/frozen cohorts remain unchanged. #809 and full tracing stay open.

    The first continuous-worker observation reaches actual workflow completion (01m52hytnc89f0tmjt7aaph6cf, 14:31:53.625455), with all 64 activities and 66 tasks completed. Its observer then fails on an undefined SIGTERM constant in the fixture's PHP build, before the full typed/prefix assertions. This is a fixture/shutdown failure, not a complete acceptance or package defect. It stays failed. A prospective fixture uses the real worker's public Looping event to request shouldQuit after canonical completion, preserving actual queue execution and the same 90-second bound; no signal or graceful-SIGTERM claim is made. A fresh original will verify the whole profile after that correction.

  14. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    Normal installed 2.5.27 business acceptance PASS on a fresh continuous-worker original 01m52j4z60xjt902mmjck5dnzg.

    • Packagist package 8ba5bec6f5fb72c6f159f4b82695f19b06017e63, PHP 8.5.11 / Laravel 13.37.0 / Testbench 11.2.0. No runtime autoload override. SQLite 3.53.4 DELETE on the host bind mount, 5,000 ms busy timeout, 2 CPU / 2 GiB / no swap.
    • Four real concurrent queue workers. The prospective lifecycle correction in 6217708710 keeps them serving until completion within the original 90-second observer. Original 120/100-second workflow deadlines and configured per-job options remain unchanged. All four exit 0 through a fixture-owned Looping stop request; this is not SIGTERM qualification.
    • All 64 activity callbacks and first-attempt activities complete once. Exactly 66 tasks and 195 events, one workflow completion, exact bigint/null/Unicode output, original whole history prefixes and complete recorded policies/deadlines pass.
    • Three repairs, zero failed jobs and zero remaining transport jobs. The separate strict zero-repair metric is FAIL, not reinterpreted as a clean pass. Delivery-exception reporting is not instrumented here, so no zero-exception claim or cause for the three repairs is inferred.
    • The normal installed focused selection separately passes 78 controls / 895 assertions, including the real post-callback competing writer and expired/replaced claim refusals. All 17 published verification jobs in 38915513763 pass.

    The earlier 45-second worker-cutoff original and undefined-SIGTERM observer original remain failed. This new completed original does not rescue or relabel them. Sample #232 candidate qualification continues. #809 and full tracing remain open for their remaining exact acceptance and hosted failure follow-through; no contention-free, capacity, native adoption or Server dependency claim.

  15. rmcdaniel commented on Oct 16, 2026

    @rmcdaniel
    MemberAuthor

    The embedded queue acceptance criteria are complete in published Workflow 2.5.27 (8ba5bec6), with the final diagnostic delivered through #818 (a51bdf64, reviewed tree e984e9d6).

    Required boundary Verified outcome
    Distinguish queued, missing and reserved transport #810's counterfactual controls cover workflow/activity/timer deliveries, stale and missing references, writer snapshots and separate transport databases.
    Bound recovery without starving genuinely missing work Small-scan and continuously arriving-backlog controls pass, preserving legacy timestamps, application connectors and existing caller transactions.
    Preserve original identity, policies, history and deadlines; callback once The actual four-worker published original completes all 64 first-attempt callbacks with exact typed output and unchanged prefix/deadlines.
    Separate claim, acknowledgment and publication pressure #812, #814 and #816 deliver the independently reproduced corrections. Real writer controls preserve original authority and refuse expiry/replacement and uncertain commit/rollback; the callback and entered publication body do not repeat.
    Enforce the original continuous-worker observer Published consumer verifies 2 CPU / 2 GiB / no swap, completes in 41.51 seconds inside the original 90 seconds, with zero repairs, failed deliveries or leftover jobs. Workers stop through Looping; no SIGTERM claim.
    Qualify source, compatible publication and installed consumers #816/#817 source and compatible gates, all 17 published profiles and 78 normal-installed controls/895 assertions pass. Sample #232 is delivered. Final #818 fresh main passes all applicable source/upgrade/database/coverage/Sonar checks; its independent main published consumer also passes.

    Earlier failed originals retain their outcomes. There is no contention-free, capacity or tracing-release claim. Draft #788 independently passes its selected 145 tracing controls but still fails the released OTel baggage case. The separately proposed PHP Server dependency/image follow-through remains tracked at Server #325; its current image/frozen references are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingintake:approvedCurrent issue title and body revision is approved for authority intakekind:defectA public product behavior is incorrectpriority:P1High-priority product or release riskstatus:in-progressApproved work is actively being implemented or validated

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions