From 708036f92bd42ed1a2dba2368104bc8c8b66d03f Mon Sep 17 00:00:00 2001 From: bidi Date: Mon, 21 Sep 2026 18:04:40 +0300 Subject: [PATCH] fixed inconsistencies with dotkernel/queue repo Signed-off-by: bidi --- docs/book/v2/control-commands.md | 34 +++++++++++++-- .../v2/how-to/communication-with-queue.md | 31 +++++++++++++ docs/book/v2/how-to/send-emails.md | 29 ++++++++++++- docs/book/v2/installation.md | 40 ++++++++++++++--- docs/book/v2/messenger-configuration.md | 43 ++++++++++++++++++- docs/book/v2/overview.md | 28 +++++++++++- docs/book/v2/server-setup.md | 35 +++++++++++++++ docs/book/v2/valkey.md | 24 +++++++++++ docs/book/v2/what-is-queue.md | 28 ++++++++++++ 9 files changed, 280 insertions(+), 12 deletions(-) diff --git a/docs/book/v2/control-commands.md b/docs/book/v2/control-commands.md index 44e862b..983f68d 100644 --- a/docs/book/v2/control-commands.md +++ b/docs/book/v2/control-commands.md @@ -1,5 +1,13 @@ # Available commands and usage +## Summary + +Three commands — `failed`, `processed`, `inventory` — report on the queue's logs and +current contents; each is available both via the local CLI and over a TCP message to +the queue server. + +## Details + The commands available are: 1. `GetFailedMessagesCommand.php (failed)` - returns logs with messages that failed to process (levelName:error) @@ -13,11 +21,11 @@ The commands can be run in two different ways: To run the commands via CLI, use the following syntax: ```shell -php bin/cli.php failed --start="yyyy-mm-dd" --end="yyyy-mm-dd" --limit=int +php bin/cli.php failed --start="yyyy-mm-dd[ HH:ii:ss]" --end="yyyy-mm-dd[ HH:ii:ss]" --limit=int ``` ```shell -php bin/cli.php processed --start="yyyy-mm-dd" --end="yyyy-mm-dd" --limit=int +php bin/cli.php processed --start="yyyy-mm-dd[ HH:ii:ss]" --end="yyyy-mm-dd[ HH:ii:ss]" --limit=int ``` ```shell @@ -29,11 +37,11 @@ php bin/cli.php inventory To use commands using TCP messages, the following messages can be used: ```shell -echo "failed --start=yyyy-mm-dd --end=yyyy-mm-dd --limit=days" | socat -t1 - TCP:host:port +echo "failed --start=yyyy-mm-dd[ HH:ii:ss] --end=yyyy-mm-dd[ HH:ii:ss] --limit=days" | socat -t1 - TCP:host:port ``` ```shell -echo "processed --start=yyyy-mm-dd --end=yyyy-mm-dd --limit=days" | socat -t1 - TCP:host:port +echo "processed --start=yyyy-mm-dd[ HH:ii:ss] --end=yyyy-mm-dd[ HH:ii:ss] --limit=days" | socat -t1 - TCP:host:port ``` In both cases, the flags are optional. Keep in mind if both `start` and `end` are set, `limit` will not be applied, it's only used when one of `start` or `end` is missing. @@ -49,3 +57,21 @@ echo "control" | socat -t1 - TCP:host:port ```shell echo "inventory" | socat -t1 - TCP:host:port ``` + +## FAQ + +**Q: What's the difference between the `failed` and `processed` commands?** + +A: `failed` returns log entries at `levelName:error` (messages that failed to +process); `processed` returns entries at `levelName:info` (messages that processed +successfully). + +**Q: Can I filter by date and also cap the number of days?** + +A: Yes, but not at the same time — `--limit` is only applied when exactly one of +`--start` or `--end` is given; if both are set, `--limit` is ignored. + +**Q: How do I quickly verify the queue is processing messages end to end?** + +A: Send the `control` message (e.g. `echo "control" | socat -t1 - TCP:host:port`); it +is always logged as processed successfully, giving you a fast round-trip check. diff --git a/docs/book/v2/how-to/communication-with-queue.md b/docs/book/v2/how-to/communication-with-queue.md index 6b3df7e..f913ed4 100644 --- a/docs/book/v2/how-to/communication-with-queue.md +++ b/docs/book/v2/how-to/communication-with-queue.md @@ -1,5 +1,12 @@ # COMMUNICATE WITH QUEUE +## Summary + +Two ways to send a message to Dotkernel Queue from your application: a quick +procedural TCP call, or a reusable service class wired into a Core module. + +## Details + Communication with the [`Dotkernel Queue`](https://github.com/dotkernel/queue) can be achieved in two different ways: procedural and object-oriented. ## Procedural approach @@ -204,3 +211,27 @@ Navigate to your handler, inject the new service and use your custom method wher protected NotificationService $notificationService ) { ``` + +> **_NOTE:_** Sending a message only queues it. `src/App/Message/MessageHandler.php` +> only acts on the literal payload values `control` and `retry` out of the box — add +> your own `elseif` branch (or replace the handler) to process the payload your +> service sends, or it will be consumed silently with no effect. + +## FAQ + +**Q: Which approach should I use — procedural or object-oriented?** + +A: Procedural is simplest for a one-off call; the object-oriented +`NotificationService` approach is better once you're sending messages from multiple +places, since it's reusable and easier to maintain. + +**Q: Why does my message need to end with a newline?** + +A: The Swoole listener uses the newline as the end-of-message marker; without it, the +server keeps waiting for more data and never processes what was sent. + +**Q: My message was accepted but nothing happened — why?** + +A: Queuing a message only stores it. `src/App/Message/MessageHandler.php` only has +explicit handling for the literal payload values `control` and `retry` out of the +box; anything else needs a handler branch you write yourself. diff --git a/docs/book/v2/how-to/send-emails.md b/docs/book/v2/how-to/send-emails.md index d672882..52af277 100644 --- a/docs/book/v2/how-to/send-emails.md +++ b/docs/book/v2/how-to/send-emails.md @@ -1,12 +1,19 @@ # SEND EMAILS +## Summary + +How to add background email sending to a Dotkernel application by importing Core +into the queue and following the `send-email` branch as a reference implementation. + +## Details + Using a queuing service solves problems such as server overload. For example, if a server receives a large number of requests, it tries to process them synchronously, resulting in long response times or even server crashes. A concrete example is sending emails. While a series of tasks are running on the server, a task such as sending an email is passed to a queue and run in the background so the server can move on to the next task, while the queue composes the email and sends it. Tasks are queued and processed gradually (FIFO), depending on available resources. To implement such a service, the [`send-email`](https://github.com/dotkernel/queue/tree/send-email) branch can be taken as a model. -> **_NOTE:_** The default branch 1.0 holds only the base code of Queue and provides essential features such as: +> **_NOTE:_** The default branch holds only the base code of Queue and provides essential features such as: > > * Adding messages to the queue > * Retrieving and processing messages (FIFO) @@ -108,3 +115,23 @@ Inside your `config/autoload` folder create a new file named `mail.global.php`, Once everything is installed and configured we can move on to handle the data in the queue. In the message handler for example `MessageHandler`, each message from the queue is processed, the email is composed, and then sent. By injecting the required services and using templates, the handler can send emails without blocking the main application, respecting FIFO and asynchronous processing. In this [file](https://github.com/dotkernel/queue/blob/send-email/src/App/Message/MessageHandler.php) you can follow a simple example of how to create and send an email using data received from the queue inside the handler. + +## FAQ + +**Q: Do I need to modify the base queue code to send emails?** + +A: No — import the `Core` module (copied in or as a submodule) and follow the +`send-email` branch as a model; the default branch already provides message queuing +and FIFO processing. + +**Q: What does importing Core actually give the queue?** + +A: Access to the main application's entities, services and configuration (cache, +mail, authentication, etc.), so the worker can compose and send real emails using +your existing templates. + +**Q: Where do I configure the mailer itself?** + +A: Create `config/autoload/mail.global.php` from the example in the `send-email` +branch and fill in your mail settings; the queue uses this to send emails in the +background. diff --git a/docs/book/v2/installation.md b/docs/book/v2/installation.md index 93a7ed7..bcc7a0c 100644 --- a/docs/book/v2/installation.md +++ b/docs/book/v2/installation.md @@ -1,5 +1,12 @@ # INSTALLATION +## Summary + +How to get a working copy of the queue running on the server prepared in +[Server setup](server-setup.md): clone the repository, configure +`config/autoload`, install dependencies, register the systemd daemons, and confirm +the listener responds. + ## Location - Because you are logged in now with the non-root user `dotkernel`, your current server path must be `/home/dotkernel` @@ -9,7 +16,7 @@ ## git clone ```shell -git clone -b default-queue https://github.com/dotkernel/queue.git +git clone https://github.com/dotkernel/queue.git ``` > The installation path should be now `/home/dotkernel/queue` @@ -17,7 +24,7 @@ git clone -b default-queue https://github.com/dotkernel/queue.git ## Prepare `config/autoload` files - duplicate `local.php.dist` as `local.php`, then fill in the database credentials and set the `$baseUrl` -- duplicate `log.local.dist` as `log.local` +- duplicate `log.local.php.dist` as `log.local.php` - duplicate `messenger.local.php.dist` as `messenger.local.php` - duplicate `swoole.local.php.dist` as `swoole.local.php` @@ -55,7 +62,7 @@ sudo systemctl start swoole.service ``` ```shell -sudo systemctl status swoole.service +sudo systemctl status swoole.service ``` ## Start the Messenger daemon @@ -73,7 +80,7 @@ sudo systemctl start messenger.service ``` ```shell -sudo systemctl status messenger.service +sudo systemctl status messenger.service ``` ### Testing the installation @@ -81,5 +88,28 @@ sudo systemctl status messenger.service Send a request from your local machine ```shell -echo "Hello" | socat -T1 - TCP:SERVER-IP:8556` +echo "Hello" | socat -T1 - TCP:SERVER-IP:8556 ``` + +> **_NOTE:_** Any message that is not one of `failed`, `processed` or `inventory` is +> queued twice by design: once with your payload, and once more with the literal +> payload `with 5 seconds delay`, queued 5 seconds later. Expect two entries in +> `inventory`/the logs for every test message you send. + +## FAQ + +**Q: Which branch should I clone?** + +A: Clone without specifying `-b`; this checks out the repository's default branch +instead of pinning to a branch name that can go stale. + +**Q: Why does copying `log.local.php.dist` correctly matter?** + +A: `config/config.php` only loads local config files that end in `.php`; if the copy +is misnamed the logger silently never loads. + +**Q: What should I see after the smoke test?** + +A: Two entries appear for the single `echo "Hello"` message you sent — your message, +plus a second, hardcoded `with 5 seconds delay` message queued automatically 5 +seconds later. diff --git a/docs/book/v2/messenger-configuration.md b/docs/book/v2/messenger-configuration.md index 8ffc624..0a4f03a 100644 --- a/docs/book/v2/messenger-configuration.md +++ b/docs/book/v2/messenger-configuration.md @@ -1,5 +1,13 @@ # Messenger Configuration +## Summary + +Reference for `config/autoload/messenger.local.php`: the Redis transports Symfony +Messenger uses for new and failed messages, their retry strategy, and the two +message streams (`messages`, `failed`) they map to. + +## Details + ```php return [ 'symfony' => [ @@ -35,7 +43,7 @@ return [ ], ], 'dependencies' => [ - 'factories'> [ + 'factories' => [ 'redis_transport' => [TransportFactory::class, 'redis_transport'], 'failed' => [TransportFactory::class, 'failed'], SymfonySerializer::class => fn(ContainerInterface $container) => new PhpSerializer(), @@ -51,3 +59,36 @@ return [ ## Dead Letter Queue (DLQ) DLQ is a dedicated transport where messages are sent when they fail to be processed after a configured number of retries. Each transport can define a retry_strategy specifying the maximum number of retry attempts, delays between retries, and exponential backoff rules. When a message exceeds the allowed retries, it is automatically forwarded to the failure transport and stored in `failed` stream, ensuring that failed messages do not block the queue. + +## Application-level retry delays (`fail-safe`) + +`config/autoload/local.php` also defines a separate `fail-safe` schedule, used to +delay re-adding a failed message to the queue: + +```php +'fail-safe' => [ + 'first_retry' => 3600000, // 1h + 'second_retry' => 43200000, // 12h + 'third_retry' => 86400000, // 24h +], +``` + +This is independent of the transport-level `retry_strategy` above. + +## FAQ + +**Q: Where do the transport-level retry settings live?** + +A: In `config/autoload/messenger.local.php`, under +`symfony.messenger.transports.redis_transport.retry_strategy`. + +**Q: What happens once `max_retries` is exceeded?** + +A: The message is forwarded to the `failed` transport, defined by +`failure_transport`, and stored in the `failed` Redis stream. + +**Q: Is `retry_strategy` the only retry configuration in the project?** + +A: No — `config/autoload/local.php` also defines an independent `fail-safe` schedule +(`first_retry`, `second_retry`, `third_retry`) for delaying re-queued messages after +a processing error. diff --git a/docs/book/v2/overview.md b/docs/book/v2/overview.md index 9c7a495..6b519c0 100644 --- a/docs/book/v2/overview.md +++ b/docs/book/v2/overview.md @@ -1,6 +1,14 @@ # Overview -> [Dotkernel Queue](https://github.com/dotkernel/dot-queue) is a component based on [**Symfony Messenger**](https://github.com/symfony/messenger) that is used to queue asynchronous tasks. +## Summary + +Dotkernel Queue is a Symfony Messenger-based component that lets Mezzio and Laminas +applications hand off slow or unreliable work to background workers instead of +processing it inline. + +## Details + +> [Dotkernel Queue](https://github.com/dotkernel/queue) is a component based on [**Symfony Messenger**](https://github.com/symfony/messenger) that is used to queue asynchronous tasks. [netglue/laminas-messenger](https://github.com/netglue/laminas-messenger) is an adapter that integrates Symfony Messenger with the [Laminas Service Manager](https://docs.laminas.dev/laminas-servicemanager/) container for Mezzio/Laminas applications. Some everyday **operations are time-consuming and resource-intensive**, so it's best if they run on separate machines, decoupled from the regular request-response cycle. @@ -23,3 +31,21 @@ It allows the main platform to return a response and remain responsive for new r [![codecov](https://codecov.io/gh/dotkernel/queue/branch/2.0/graph/badge.svg?token=pexSf4wIhc)](https://codecov.io/gh/dotkernel/queue) [![Qodana](https://github.com/dotkernel/queue/actions/workflows/qodana_code_quality.yml/badge.svg?branch=2.0)](https://github.com/dotkernel/queue/actions/workflows/qodana_code_quality.yml) [![PHPStan](https://github.com/dotkernel/queue/actions/workflows/static-analysis.yml/badge.svg?branch=2.0)](https://github.com/dotkernel/queue/actions/workflows/static-analysis.yml) + +## FAQ + +**Q: What is Dotkernel Queue built on?** + +A: It's based on Symfony Messenger, integrated into Mezzio/Laminas applications through +the `netglue/laminas-messenger` adapter for the Laminas Service Manager container. + +**Q: Why run tasks asynchronously instead of inline?** + +A: Time-consuming or resource-intensive operations would otherwise block the +request-response cycle; running them on background workers keeps the main platform +responsive to new requests. + +**Q: Where can I find the project's build and license status?** + +A: See the badges above, which link to the GitHub issues, forks, stars, license, CI, +code coverage, and static analysis pages for the `dotkernel/queue` repository. diff --git a/docs/book/v2/server-setup.md b/docs/book/v2/server-setup.md index 8fea652..e9c7db7 100644 --- a/docs/book/v2/server-setup.md +++ b/docs/book/v2/server-setup.md @@ -1,5 +1,13 @@ # Server setup +## Summary + +Step-by-step instructions for provisioning a fresh AlmaLinux 9/10 server with the +users, PHP runtime, Swoole and Redis/Valkey extensions, and firewall rules the queue +daemon needs. + +## Details + The below instructions were tested only on **AlmaLinux 9** or **10**. *For other operating systems, they need to be adapted accordingly.* @@ -20,6 +28,7 @@ dnf update -y ```shell useradd dotkernel +useradd --system --no-create-home queue ``` ```shell @@ -54,6 +63,9 @@ sudo dnf install -y https://rpms.remirepo.net/enterprise/remi-release-$(rpm -E % sudo dnf module enable php:remi-8.5 ``` +> PHP 8.4 (`php:remi-8.4`) is also supported (`composer.json` allows +> `~8.4.0 || ~8.5.0`); substitute the module version above if you need 8.4. + ```shell sudo dnf install -y php php-cli php-common php-intl ``` @@ -166,3 +178,26 @@ sudo firewall-cmd --reload ``` > NOW THE SERVER IS READY + +## FAQ + +**Q: Which operating systems does this guide support?** + +A: It was tested on AlmaLinux 9 and 10; other operating systems need the steps +adapted accordingly. + +**Q: Which PHP versions can I install?** + +A: PHP 8.5 (`php:remi-8.5`) is documented here, and PHP 8.4 (`php:remi-8.4`) is also +supported since `composer.json` allows `~8.4.0 || ~8.5.0`. + +**Q: Which system users does the queue need?** + +A: A sudo-capable `dotkernel` user for administration, and a `queue` system +user/group, which is what the shipped `swoole.service` and `messenger.service` unit +files run as. + +**Q: Is the firewall setup mandatory?** + +A: No, but it's recommended — it restricts inbound connections on the queue's TCP +port (8556 by default) to specific source IPs. diff --git a/docs/book/v2/valkey.md b/docs/book/v2/valkey.md index e6a0c08..cab47cb 100644 --- a/docs/book/v2/valkey.md +++ b/docs/book/v2/valkey.md @@ -1,5 +1,12 @@ # Valkey usage +## Summary + +Quick reference for the Valkey CLI commands used to inspect and manage the +`messages` and `failed` streams that back the queue. + +## Details + Valkey is an open source (BSD) high-performance key/value datastore that supports a variety of workloads such as caching, message queues and can act as a primary database. The following commands can be run in the CLI to interact with Valkey. @@ -86,3 +93,20 @@ Delete a specific entry: ```shell XDEL streamName ``` + +## FAQ + +**Q: How do I open an interactive Valkey session?** + +A: Run `valkey-cli` on the server; it drops you into the CLI used for every command +on this page. + +**Q: How do I read the queue's stream without removing anything?** + +A: Use `XRANGE streamName - +` to read entries oldest to newest (switch `-`/`+` to +reverse the order); it doesn't delete anything. + +**Q: How do I clear a stream without deleting the key itself?** + +A: `XTRIM streamName MAXLEN 0` removes all entries but keeps the stream key, unlike +`DEL streamName` which removes the key entirely. diff --git a/docs/book/v2/what-is-queue.md b/docs/book/v2/what-is-queue.md index 3d2ddc1..b92d5a5 100644 --- a/docs/book/v2/what-is-queue.md +++ b/docs/book/v2/what-is-queue.md @@ -1,5 +1,12 @@ # Tasks Delegated to the Queue +## Summary + +What kinds of tasks belong in the queue, how the queue processes them, and the core +features (logging, security, retries, reporting) that make it reliable. + +## Details + Normally, the tasks delegated to the queue: - Take extended periods of time to execute and may be interrupted by PHP limitations (like the PHP `max_execution_time` parameter). @@ -38,3 +45,24 @@ The order of the execution uses the **FIFO** (First-In, First-Out) method where - Processing time per job. - Error rates - How many messages failed. - Throughput - Jobs/sec processed. + +## FAQ + +**Q: What kinds of tasks should be delegated to the queue?** + +A: Long-running, external, or non-response tasks such as data processing, file/media +processing, networking, database operations, and system/infrastructure tasks. + +**Q: In what order does the queue process messages?** + +A: FIFO (First-In, First-Out) — the oldest request is processed first, followed by +newer requests. + +**Q: How does the queue protect against untrusted senders?** + +A: A firewall allows requests only from whitelisted IPs. + +**Q: What happens when a task fails?** + +A: The retry mechanism retries failing tasks a certain number of times before removing +the task from the queue.