Skip to content

Add firefly/orchestration: sagas, TCC and durable workflows - #15

Merged
ancongui merged 59 commits into
mainfrom
feat/orchestration
Oct 13, 2026
Merged

ancongui merged 59 commits into
mainfrom
feat/orchestration

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Adds firefly/orchestration: sagas, TCC and durable workflows for LaraFly, modeled on jafly's fireflyframework-orchestration and at parity with PyFly's transactional module.

What ships

  • Sagas. Steps are declared with #[Saga] and #[SagaStep(dependsOn, compensate, retry, backoffMs, timeoutMs, idempotencyKey, …)].
    • Steps are organized in topology layers. Steps within a layer run sequentially in declaration order, because PHP is synchronous; layer semantics are preserved.
    • Compensation follows one of five policies (strict-sequential, grouped, retry-with-backoff, circuit-breaker, best-effort), with pluggable error handlers.
    • A compensation that stops partway ends COMPENSATION_FAILED and is dead-lettered.
    • Also included: a SagaBuilder DSL, saga composition with data flow, argument injection (#[Input], #[FromStep], #[Header], #[Variable], #[CorrelationId]), #[StepEvent], completion and error callbacks, and #[ScheduledSaga].
  • TCC. Ordered #[TccParticipant]s with try/confirm/cancel methods, per-phase timeouts and retries, and #[FromTry]. A TRY that times out is cancelled. TCC executions can be composed.
  • Durable workflows.
    • Steps support dependencies and conditions (Symfony ExpressionLanguage).
    • Waiting: signals, timers, wait-for-all/any, child workflows.
    • Lifecycle: continue-as-new, queries, suspend/resume, search attributes, compensation steps.
    • Workflows run SYNC or ASYNC on a Laravel queue.
    • Due work is resumed by firefly:orchestration:tick, which also runs as a #[Scheduled] task under the DistributedLock port.
    • A poison execution is parked and dead-lettered after firefly.orchestration.workflow.max-resume-attempts, so it cannot hold up the tick.
  • Persistence. firefly.orchestration.persistence.provider=memory|cache|database.
    • The database provider uses portable tables and is integration-tested on PostgreSQL, MySQL, MariaDB, SQL Server and SQLite.
    • Each execution is guarded by a per-execution lock.
    • Also: a recovery sweep (report, resume or compensate stale executions), retention cleanup, and a dead-letter queue (memory|database) with retry.
  • Operations.
    • Lifecycle and step events go out through firefly/eda, and executions can be started by event-type triggers.
    • Observability: an orchestration actuator endpoint and health component, plus metrics and spans through firefly/observability.
    • Commands: php artisan firefly:orchestration:{list,show,signal,dlq,dlq:retry,tick}.
    • An opt-in REST surface.
    • Topology is validated at boot (cycles, unknown dependencies, missing compensations) and fails fast.

Notes

  • Includes origin/main (wave W1). The integration tests use IntegrationDsn and its lanes, so they run in the new CI matrix.
  • SQL Server fix: the search-attribute upsert ran as a table-scanning MERGE, which blocked on other workers' uncommitted rows. It now writes by exact primary key.

Verification

Run Result
composer check exit 0, 6255 passed
Integration on PostgreSQL 16, MySQL 8.4, MariaDB 11 and SQLite 190 passed, 0 failed
SQL Server lane (local, Azure SQL Edge) 60 passed, 0 failed

Andres Contreras added 30 commits October 12, 2026 18:52
… and refuse non-finite floats and invalid UTF-8
…ache providers, and a re-entrant execution lock
… the dead-letter queue and back-pressure admission
…idempotency, compensation, dead letters, resume
…ot undo and tolerate optional members that cannot start
…phase retries, phase checkpoints and composition
… a resume never retries it or reports CONFIRMED
… resume an interrupted cancel on request as one
…waits, children and queries, and the builder DSL
… time and refuse a manifest wait without a known kind
…, continue-as-new and queries, plus dead-letter retry
… re-driven by its children, and keep the tick going past an execution it cannot drive
… keep the continue-as-new origin out of children, fail a parent whose child cannot start, and refuse a taken workflow correlation id
…nued as new still wakes its parent, keep the report and deadline on a search-attribute upsert, back off due workflows the tick cannot drive, and lock the ASYNC start
…ore, proven on PostgreSQL, MySQL, MariaDB and SQLite
Andres Contreras added 28 commits October 12, 2026 21:24
…e definitions at boot, and start executions from EDA triggers
…the event id and carry the envelope headers, so a redelivered event is acknowledged instead of starting the definition again
…utions, the scheduled tick and sweep, and scheduled orchestrations
…covery instead of driving the workflow forward
…recovery acts, so a run that finished or checkpointed while the sweep waited is skipped instead of undone or driven twice
… and the checkpoint in wakeAt, so the tick still wakes that waiting workflow
…mponent, and commands to list, show, signal, tick and retry
…he scanner, the capstone and on every database
…MPENSATION_FAILED, dead-letter it and name the steps it left uncompensated
…esume-attempts times FAILED and dead-letter it, and let database faults escape the tick
…and correlation id, and keep the error message at debug level
…ow by depth, so a nested call no longer clears the outer call's mark
…ompensation held back, so the parent does not stay WAITING on a child it already cancelled
# Conflicts:
#	CHANGELOG.md
#	README.md
#	composer.json
#	docs/README.md
#	docs/book.md
#	docs/index.md
#	docs/modules.md
#	samples/lumen/tests/Integration/OutboxPostgresIntegrationTestCase.php
… run every lane of the CI matrix

IntegrationDatabase now delegates to Firefly\Testing\Integration\IntegrationDsn instead of parsing
FIREFLY_* variables by hand, iterates IntegrationDsn::lanes() (the four servers plus a sqlite file) and
skips with IntegrationDsn::skipReason(), so each CI integration job runs its own lane. The two-worker
interleaving test runs on the server lanes only, and sets SQL Server's LOCK_TIMEOUT unprepared, since a
prepared SET is undone when its statement ends.
…a save on SQL Server no longer waits on another execution's open checkpoint

The multi-row upsert ran as a MERGE that SQL Server can execute as a scan of the whole attributes
table, and under READ COMMITTED that scan blocked on the rows another worker's uncommitted save held.
save() now updates each kept key and inserts the new ones by primary key, touching only the
execution's own rows on every server.
…ation lane, and add orchestration to the database support matrix
… so no test inherits another's session settings
…at shortens it, since the runner's driver manager pools sessions
@ancongui
ancongui merged commit 5aa58b4 into main Oct 13, 2026
12 checks passed
@ancongui ancongui mentioned this pull request Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant