CI pulls redis:7.0.15 from Docker Hub unauthenticated; since #34 a Hub outage or rate limit blocks deploys #43

Activity

1. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Hand-off from the burn-down epic #24, 2026-10-10. This is next, ahead of #5. Under the epic's ordering rule, deploy safety comes before agent tooling. #39 shipped in #42.

   Why now. Since #34, scripts/deploy.sh deploys only a commit whose main push run is green, and Processor has decided production results since watcher#326. On 2026-10-09, CI's Redis service image failed to pull twice (Docker Hub's anonymous rate limit, then a Hub incident), which blocked #39's merge and deploy until re-runs. Small, CI only, no deploy.

   Touchpoint: .github/workflows/ci.yml, job test (lines 72–95): runs-on: ubuntu-latest, services.redis.image: redis:7.0.15 with a health check. Its comment (lines 79–84) explains why 7.0.15 and not 7.2 (redis-py's CLIENT SETINFO versus the broker's ACL). That constraint stays.

   The likely fix, measured here: Ubuntu 24.04's own apt redis-server is 7.0.15. On co-processor (24.04.5), apt-cache policy redis-server gives 5:7.0.15-1ubuntu0.24.04.4, and that's the scratch server the integration suite already runs against locally. So: install it from apt on the runner and drop the Docker Hub image. Ubuntu's mirrors become the dependency instead of anonymous Hub pulls.

   Traps:

   1. ubuntu-latest moves. When GitHub points it at the next LTS, apt's Redis jumps past 7.0 and the CLIENT SETINFO constraint breaks silently. Pin runs-on: ubuntu-24.04 for the test job, and add a step that asserts redis-server --version is 7.0.15 (fail loudly, never test against a different Redis).
   2. The suite reconfigures Redis at runtime: db 15, a throwaway ACL user, and briefly maxmemory (the comment's "all of which a service container absorbs"). An apt install starts a systemd redis-server with /etc/redis/redis.conf (bind 127.0.0.1, protected-mode yes). Check that CONFIG SET maxmemory and ACL SETUSER work against it as they do against the container, and that nothing in the stock config (e.g. rename-command) differs. Starting one directly (redis-server --port 6379 --daemonize yes --save '' --appendonly no) avoids the stock config. Pick one and say why.
   3. Keep the health gate. The service block waited for redis-cli ping. A started daemon needs the same bounded wait before pytest.
   4. apt-get update adds time and a network step. Weigh apt-get install without update (the runner image's package index may already have it) against reliability. Measure the job time before and after.

   Alternatives (from the issue, if apt fails a trap): mirror.gcr.io/library/redis:7.0.15 (Google's Hub mirror, no credentials), or an authenticated pull (a secret to manage). Record why you didn't choose them.

   Done when: CI's test job no longer pulls from Docker Hub anonymously; it asserts Redis 7.0.15; the integration suite passes in full (same counts as before: compare the test totals with the last green run); and job time is reported. No deploy, because .github/ isn't a runtime path.

   Ship: issue → branch in a worktree → PR → FF merge → git pull --ff-only. The closing comment gives the before and after job times and test counts.

   Kickoff prompt:

   Pick up processor#43. Read its hand-off comment from epic #24 first.
2. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Shipped in #44 (FF merge, f36cebc). No deploy: .github/, tests/ and AGENTS.md aren't runtime paths.

   Choice: Google's Hub mirror, as Watcher did (watcher 61be8bd2, tests/ci/test_service_images.py), not the hand-off's apt route. The test job now pulls mirror.gcr.io/library/redis:7.0.15. The mirror and Docker Hub return the same index digest (sha256:352c1fdadc91926edda08f45aeb3f27f37194c2f14101229c0523a11195c96e3), so this is the same image, with no anonymous limit and no credential. The service block, health gate and 7.0.15 pin are unchanged.

   • Why not apt: it needs a pinned ubuntu-24.04, a version assert, a hand-written health wait and an apt-get network step, and its runtime config differs from the container's (traps 1–4). The mirror changes only the registry, so the Redis the suite runs against stays the same.
   • Why not an authenticated pull: it adds a secret to manage and gains nothing over the mirror.
   • Guard: tests/test_ci.py pins BROKER_REDIS to the mirror image and sweeps every workflow's services:/container: images for the mirror.gcr.io/ prefix. A second test checks that the sweep finds ci.yml:test:redis, so it can't pass on an empty list.

   	before: run 37996667935 (fe0de17)	after: run 38009354576 (f36cebc, main push)
   test job	3m14s	2m44s
   Initialize containers	9s	13s
   pytest	577 passed, 13 skipped (85.9s)	579 passed, 13 skipped (80.4s)

   The +2 are the new tests. The job-time difference is mostly the wheelhouse sync's run-to-run variance. Pulling from the mirror costs ~4s at container init (11–13s across three runs).

   Unrelated flake on the PR's first run (38008871788): tests/test_liveness.py::test_ok_is_logged_on_the_first_and_after_a_failure_only failed [20, 20] == [20, 20, 20], and its last tick logged "previous check-in still in flight". Cause: the check-in thread settles its future with call_soon_threadsafe and only exits afterwards, so is_alive() stays true for a moment after the tick returns. In this test, the stale tick before the final one doesn't await anything, so the final tick can see the previous thread still alive. With the 300s interval this has no effect in production. It passed on re-run. Worth its own issue (#39 follow-up).
3. gregoryfoster

   closed this as completedon Oct 10, 2026
4. gregoryfoster

   added 2 commits that reference this issue on Oct 10, 2026

   #43 docs: CR 5 — the 7.0.15 pin points at broker#85

   ...

   97fc2d9

   #43 test: CR 6 — the mirror check's message states its rule

   ...

   2ecb9d4
5. gregoryfoster commented on Oct 10, 2026

   

   gregoryfoster

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Review fixes shipped in #46 (FF merge, e6febd3..2ecb9d4), two CR --fix rounds on #44. main push run 38073168716 is green: 580 passed, 13 skipped. No deploy, since nothing here runs in production.

   • CR 1: ci.yml names the mirror's failure mode. mirror.gcr.io serves only tags it has cached (old tags 404), so "manifest unknown" at Initialize containers means 7.0.15 dropped out of the cache, not a broken commit. The comment names the fallback: Ubuntu 24.04's apt redis-server 7.0.15.
   • CR 2, 3, 6: the image check also covers uses: docker://<image> steps, parses the workflows once, and its failure message states the rule ("not on mirror.gcr.io/").
   • CR 5: the 7.0.15 pin (in ci.yml, BROKER_REDIS and AGENTS.md) points to Redis 7.0 is upstream EOL (2024-07-29): decide the broker's next Redis broker#85. Redis 7.0 is upstream EOL (2024-07-29), and the broker's next Redis is decided there. This pin, the CI image and co-processor's scratch Redis move with it.

   Follow-ups: #45, the liveness test's in-flight race. It failed on CI twice in two days, and a red main run blocks deploys. Also CannObserv/broker#85.