From 78691dfec041c196d84e337f850ffcf46f88b193 Mon Sep 17 00:00:00 2001 From: Yasin Gunaydin <81875281+yasingunaydiin@users.noreply.github.com> Date: Sat, 7 Nov 2026 22:56:43 +0100 Subject: [PATCH 1/2] Add Emergency Rescue investigation page Add a troubleshooting page that explains what an Emergency Rescue is, why it shows up in the Control Panel logs and livelog, and how to use the incident files and PHP-FPM log to find out what caused it. Link to it from the 24/7 Monitoring and Alerting page. --- .../support/24-7-monitoring-and-alerting.md | 2 +- ...-on-hypernode-and-how-to-investigate-it.md | 78 +++++++++++++++++++ 2 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md diff --git a/docs/about-hypernode/support/24-7-monitoring-and-alerting.md b/docs/about-hypernode/support/24-7-monitoring-and-alerting.md index 760dfeea..dd4e966e 100644 --- a/docs/about-hypernode/support/24-7-monitoring-and-alerting.md +++ b/docs/about-hypernode/support/24-7-monitoring-and-alerting.md @@ -17,7 +17,7 @@ At Hypernode we understand like no other that the availability and stability of ## Monitoring -It is our responsibility to monitor all the offered services on the Hypernode. As such we have setup a global monitoring network that checks your Hypernode’s availability from locations in Europe, the United States, and Eastern Asia. Each location checks the availability roughly every 2 minutes. An alert is raised if too many errors are received globally, and an Emergency Rescue is started. If the automated Emergency Rescue can’t fix the issue, the alert is escalated to the Hypernode Emergency Recovery Operative (Hero). +It is our responsibility to monitor all the offered services on the Hypernode. As such we have setup a global monitoring network that checks your Hypernode’s availability from locations in Europe, the United States, and Eastern Asia. Each location checks the availability roughly every 2 minutes. An alert is raised if too many errors are received globally, and an Emergency Rescue is started. If the automated Emergency Rescue can’t fix the issue, the alert is escalated to the Hypernode Emergency Recovery Operative (Hero). See [Emergency Rescue on Hypernode and How to Investigate It](../../troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md) for what a rescue does and how to find out what caused it. ### What Do We Monitor diff --git a/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md b/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md new file mode 100644 index 00000000..952e990c --- /dev/null +++ b/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md @@ -0,0 +1,78 @@ +--- +myst: + html_meta: + description: Learn what an Emergency Rescue on Hypernode is, why it shows up in + your logs, and how to use the incident files to find out what caused it. + title: Emergency Rescue on Hypernode and How to Investigate It | Hypernode +--- + +# Emergency Rescue on Hypernode and How to Investigate It + +An Emergency Rescue is started automatically when our global monitoring sees too many errors on your Hypernode. It tries to fix the problem without anyone having to act, usually by restarting services that fail under heavy load. If it can't, the alert goes to the Hypernode Emergency Recovery Operative (Hero). See our guide on [24/7 monitoring and alerting](../../about-hypernode/support/24-7-monitoring-and-alerting.md) for how this works. + +You do not need to do anything when a rescue is started. + +## Why It Shows Up With Your Name + +In the logs of the Control Panel you may see "Emergency Rescue" entries with your name, or the name of the account owner, and an IP of `None` with source `Hypernode Events`. Nobody logged in. The entry is created by the automated system and uses the account owner's name for the audit trail. + +## The Livelog Line + +In `livelog` or the Control Panel logs you may see a line like `restore from backup `. This is how an Emergency Rescue shows up. It is an action by the platform. Nothing in the Control Panel or in your shop started it, and it does not mean an attacker or a team member did something. + +The line says nothing about your data. To check that nothing was rolled back, look at the data itself. For example, an order placed after the time of the rescue should still be there. + +## What It Does and Does Not Look At + +- It checks if your Hypernode services (Nginx, MySQL, Redis, Varnish and RabbitMQ) are available and responding. +- It does not look at your application. It does not trigger on slow PHP requests, so it can't be set to start at a PHP response time of, for example, 4 seconds. Use an APM tool such as [Tideways](../../best-practices/performance/how-to-set-up-tideways.md), [Blackfire](../../best-practices/performance/how-to-use-blackfire-on-hypernode-to-find-performance-issues.md) or [New Relic](../../best-practices/performance/how-to-find-your-performance-bottleneck-with-new-relic.md) for that. +- A rescue can start without needing to change anything. This happens when the errors are gone again by the time it checks. + +## Services Restarting Without a Crash + +After a rescue you may see a service, for example MySQL, stop and start again in the system logs without an out-of-memory kill or a crash. This is the rescue restarting it. Several rescues in a row point to overload that lasts, often caused by bots. + +## Find Out What Caused It + +For every rescue, Hypernode saves a snapshot of the state of your Hypernode in a folder named after the time of the rescue: + +```console +app@abcdef-example-magweb-cmbl:~$ ls /data/web/incidents +``` + +```{important} +The folder names are in UTC. Convert the time shown in the Control Panel to UTC first, or you will open the wrong folder. +``` + +Each folder contains files like these: + +- `fpm-status`: the PHP-FPM workers and what they were doing. `BUSY` workers show requests that were running. +- `mysql-process-list`: the MySQL queries that were running. +- `process-list`: the running processes. +- `tcp-connections`: the open network connections. + +### Most Common Cause: PHP-FPM Is Full + +Most rescues we see are caused by all PHP-FPM workers being busy. New requests then have to wait, and the monitoring sees errors. Check the PHP-FPM log: + +```console +app@abcdef-example-magweb-cmbl:~$ grep "reached pm.max_children" /var/log/php-fpm/php-fpm.log +``` + +If you see lines here shortly before the rescue, the workers were all in use. The usual causes are: + +- Bots and crawlers requesting uncacheable pages, such as layered navigation filter URLs. See our guide on [bots and crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md). +- Scanners requesting hundreds of paths that don't exist. See our guide on [blocking spammers and scanners](../../hypernode-platform/nginx/how-to-block-spammers-and-scanners.md). +- An integration that requests your REST API very often. +- Slow requests or slow database queries in your own code. An APM tool helps to find these. + +To deal with unwanted traffic, work in this order: + +1. Return `410 Gone` to bots on layered navigation query strings. See [How to Fix Performance Issues Caused by Bots and Crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md). +1. [Rate limit](../../hypernode-platform/nginx/how-to-resolve-rate-limited-requests-429-too-many-requests.md) the crawler that causes most of the requests. +1. Filter the rest with [Botstopper](../../hypernode-platform/botstopper/how-to-use-botstopper.md). +1. Only then look at capacity. + +Check the access log afterwards to confirm the `410` and `429` responses are given. + +The number of PHP workers is fixed by your plan. If your shop needs more headroom after you have dealt with unwanted traffic, a larger plan gives you more. From 713cb5562f35e81242d56a9b255a40b57658edf7 Mon Sep 17 00:00:00 2001 From: Yasin Gunaydin <81875281+yasingunaydiin@users.noreply.github.com> Date: Mon, 9 Nov 2026 22:42:26 +0100 Subject: [PATCH 2/2] Update Emergency Rescue page Show the emergency_rescue livelog line, explain that incident snapshots are only saved for OOM events, link to livefpm, explain that more PHP-FPM workers don't help, and remove the Blackfire mention. --- ...cue-on-hypernode-and-how-to-investigate-it.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md b/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md index 952e990c..869e6a16 100644 --- a/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md +++ b/docs/troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md @@ -18,14 +18,12 @@ In the logs of the Control Panel you may see "Emergency Rescue" entries with you ## The Livelog Line -In `livelog` or the Control Panel logs you may see a line like `restore from backup `. This is how an Emergency Rescue shows up. It is an action by the platform. Nothing in the Control Panel or in your shop started it, and it does not mean an attacker or a team member did something. - -The line says nothing about your data. To check that nothing was rolled back, look at the data itself. For example, an order placed after the time of the rescue should still be there. +In `livelog` or the Control Panel logs you may see a line like `emergency_rescue`. This is how an Emergency Rescue shows up. It is an action by the platform. Nothing in the Control Panel or in your shop started it, and it does not mean an attacker or a team member did something. ## What It Does and Does Not Look At - It checks if your Hypernode services (Nginx, MySQL, Redis, Varnish and RabbitMQ) are available and responding. -- It does not look at your application. It does not trigger on slow PHP requests, so it can't be set to start at a PHP response time of, for example, 4 seconds. Use an APM tool such as [Tideways](../../best-practices/performance/how-to-set-up-tideways.md), [Blackfire](../../best-practices/performance/how-to-use-blackfire-on-hypernode-to-find-performance-issues.md) or [New Relic](../../best-practices/performance/how-to-find-your-performance-bottleneck-with-new-relic.md) for that. +- It does not look at your application. It does not trigger on slow PHP requests, so it can't be set to start at a PHP response time of, for example, 4 seconds. Use an APM tool such as [Tideways](../../best-practices/performance/how-to-set-up-tideways.md) or [New Relic](../../best-practices/performance/how-to-find-your-performance-bottleneck-with-new-relic.md) for that. - A rescue can start without needing to change anything. This happens when the errors are gone again by the time it checks. ## Services Restarting Without a Crash @@ -34,7 +32,7 @@ After a rescue you may see a service, for example MySQL, stop and start again in ## Find Out What Caused It -For every rescue, Hypernode saves a snapshot of the state of your Hypernode in a folder named after the time of the rescue: +When a rescue is caused by an [out-of-memory (OOM) event](how-to-debug-out-of-memory-oom-events.md), Hypernode saves a snapshot of the state of your Hypernode in a folder named after the time of the rescue: ```console app@abcdef-example-magweb-cmbl:~$ ls /data/web/incidents @@ -51,7 +49,7 @@ Each folder contains files like these: - `process-list`: the running processes. - `tcp-connections`: the open network connections. -### Most Common Cause: PHP-FPM Is Full +### Most Common Cause: PHP-FPM Workers Are Full Most rescues we see are caused by all PHP-FPM workers being busy. New requests then have to wait, and the monitoring sees errors. Check the PHP-FPM log: @@ -59,7 +57,9 @@ Most rescues we see are caused by all PHP-FPM workers being busy. New requests t app@abcdef-example-magweb-cmbl:~$ grep "reached pm.max_children" /var/log/php-fpm/php-fpm.log ``` -If you see lines here shortly before the rescue, the workers were all in use. The usual causes are: +If you see lines here shortly before the rescue, the workers were all in use. To see what the PHP-FPM workers are doing right now, use [livefpm](../../hypernode-platform/tools/what-kind-of-request-is-my-hypernode-serving.md#livefpm). + +The usual causes are: - Bots and crawlers requesting uncacheable pages, such as layered navigation filter URLs. See our guide on [bots and crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md). - Scanners requesting hundreds of paths that don't exist. See our guide on [blocking spammers and scanners](../../hypernode-platform/nginx/how-to-block-spammers-and-scanners.md). @@ -75,4 +75,6 @@ To deal with unwanted traffic, work in this order: Check the access log afterwards to confirm the `410` and `429` responses are given. +Increasing the number of PHP-FPM workers does not solve this. The extra workers fill up with the same requests, and they use more CPU and memory. + The number of PHP workers is fixed by your plan. If your shop needs more headroom after you have dealt with unwanted traffic, a larger plan gives you more.