Skip to content

Make the outbox, scheduling locks and CI database-agnostic - #14

Merged
ancongui merged 34 commits into
mainfrom
feat/adapter-agnostic
Oct 13, 2026
Merged

ancongui merged 34 commits into
mainfrom
feat/adapter-agnostic

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

LaraFly's same-transaction outbox, scheduling locks and CI no longer depend on PostgreSQL alone.

What changes

  • firefly/eda-database (firefly.eda.provider=database): the outbox on any Laravel connection. Rows are claimed by a lease committed before delivery, through an OutboxClaimStrategy chosen from the server version: SKIP LOCKED on PostgreSQL, MySQL 8.0.1+ and MariaDB 10.6+, READPAST on SQL Server, and a portable optimistic lease elsewhere (SQLite, unknown drivers). It can be overridden with firefly.eda.database.claim-strategy.
  • firefly/eda-postgres is now the database outbox plus pg_notify/LISTEN. The old classes and firefly.eda.postgres.* keys still work as deprecated subclasses and aliases. The outbox table gains claimed_by / claimed_until (upgrade migration included).
  • Durable dead letters for eda and messaging (...dead-letter.store=database), with replay commands.
  • firefly/scheduling-database: lock.provider=database (a portable lease table) and lock.provider=mysql (GET_LOCK, MySQL and MariaDB).
  • CI:
    • an integration job matrix over PostgreSQL 16, MySQL 8.4, MariaDB 11 and SQL Server 2022;
    • a brokers job for RabbitMQ, Kafka and Redis;
    • composer test:integration and docker-compose.integration.yml to run the same lanes locally.
  • Data layer on real servers: exception translation, statement timeouts (including a new SQL Server LOCK_TIMEOUT branch), and pessimistic/optimistic locking.
  • Docs: README, index, the module pages and diagrams no longer present PostgreSQL as the only atomic outbox. New pages: eda-database, scheduling-database, integration-testing and a database support matrix.

Fixed

  • Concurrent consumers could deliver an outbox row twice. The Postgres consumer's FOR UPDATE SKIP LOCKED lock ended with its statement, and on other drivers the claim was an unlocked read. Two racing processes are now integration-tested to deliver every row exactly once on each server.
  • The RabbitMQ round trip against a real broker, and the samples/lumen integration test that called a non-existent firefly:eda:consume command.
  • IntegrationDsn never echoes a DSN (and its password) in errors, validates SQL Server certificates by default, creates SQLite test files in a private 0700 directory, and does not let a URL query override host or credentials.

Verification

  • composer check (Pint, PHPStan max, Pest, Deptrac): exit 0, 5877 tests passed.
  • pest --group integration against real PostgreSQL 16, MySQL 8.4, MariaDB 11, RabbitMQ, Kafka and Redis: 124 passed, 0 failed. The 16 skips are SQL Server tests, which run only in the new CI lane because there is no arm64 image locally.

Andres Contreras added 30 commits October 12, 2026 18:10
…fixing the RabbitMQ URL form and the lumen consume command
…ainst the brokers, mirrored locally by docker-compose.integration.yml
…pter packages and register them everywhere
…upgrade migration; eda-postgres keeps the old names as deprecated subclasses
…DPAST and portable optimistic-lease claims, chosen by server capability
…ny connection; PostgresEventPublisher adds only the in-transaction pg_notify
…very so any number of consumers is safe; PostgresEventConsumer adds LISTEN/NOTIFY
…ys is safe; settings read firefly.eda.database.* with the postgres keys as aliases
…outbox — writer, pre-commit hook, leased consumer and an outbox health indicator
… pg_notify and LISTEN, reading the shared outbox settings
…ox row exactly once on PostgreSQL, MySQL, MariaDB, SQL Server and SQLite
…refly.eda.dead-letter.store, and firefly:eda:dead-letters:replay
…ind firefly.messaging.dead-letter.store, and firefly:messaging:dead-letters:replay
…t, so a long or non-UTF-8 key no longer fails the insert
…aw bytes and millisecond instants on every server
…DistributedLock behind firefly.scheduling.lock.provider=database
… MariaDB behind firefly.scheduling.lock.provider=mysql
…store it, proven with the other drivers' timeouts on real servers
…rt matrix, the integration lanes, and the W1 changelog
…ay oldest first, keyed by monotonic UUIDv7 ids
…ot only Postgres, with same-transaction atomicity
… SQL Server certificates by default, and create sqlite test files privately
@ancongui
ancongui merged commit ee68e17 into main Oct 13, 2026
12 checks passed
@ancongui ancongui mentioned this pull request Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant