[pithead-developer]

What's the problem

Closes #362

A DynamicUser cannot open the sister API or control receiver entrypoint beneath a 0750 home checkout. The sister API can remain unavailable while apply reports success. This supplies the RigForge regression required by p2pool-starter-stack/pithead#3400.

How I fixed it

Load each Python entrypoint through systemd credentials alongside its config, preserving the unprivileged service sandbox and home permissions. Return sister-API rendering, startup, refresh and bounded listener failures from apply while completing later reconciles.

The stock-home regression now invokes real sudo ./rigforge.sh setup before any apply (tests/test-stock-home.sh:59). It starts without a worker build or miner unit, installs dependencies and compiles pinned XMRig, verifies the real binary and unit (:62), and requires authenticated port 8081 while HOME remains 0750 (:89). A closed loopback pool and donation zero supply no mining jobs. The real miner is stopped after setup, and real CLI uninstall runs on exit. The current case retains missing-entrypoint apply failure and recovery checks.

The control harness now uses the existing bounded unauthenticated readiness probe before its first POST (tests/e2e-pithead-control.sh:41), replacing a fixed three-second sleep. Delayed and unavailable listener cases assert that no POST occurs before readiness and that failed setup restores the config (tests/test-refresh-window-control.sh:22).

Historical and current setup cases run on separate fresh Ubuntu 24.04 hosted runners (.github/workflows/ci.yml:128). Keep the hardware-harness placement preflight, the focused failure regressions and bounded control-startup diagnostics; correct the testing documentation.

Here is the proof

Head: b50bc7eddab80ceb72cf0ecf350e77848c74643c. All ten exact-head GitHub checks pass, including the real setup historical/current matrix, coverage and Docker e2e. Exact-head control job 3025 passes. This PR remains draft for independent reviewer handoff.

Review RETURN addressed: the empty-build/fake-unit apply fixture was replaced by ordinary documented setup, with authenticated listener assertions before apply (tests/test-stock-home.sh:59, :62, :89; registered at .github/workflows/ci.yml:128). Both cases passed again in exact-head run 40400405850:

• Historical RED: timeout 1800 bash tests/test-stock-home.sh b2d4c3d9ca1da74d0e6d5ccce772d401acc8be03 broken (job log): stock-home setup historical RED: real setup completed but the sister entrypoint was denied and port 8081 unavailable.
• Fixed GREEN: timeout 1800 bash tests/test-stock-home.sh HEAD working (job log): stock-home setup GREEN: real setup built XMRig, installed its unit, and served authenticated port 8081 beneath a 0750 HOME, followed by stock-home apply GREEN: failed reconcile rejected, service recovered, HOME remains 0750.

These runs install dependencies, build XMRig and install real units on fresh Ubuntu 24.04 as a normal user, without stubs, a prebuilt worker or appliance mode. Both quoted results are from the current PR head; the earlier 4cadfaa matrix also passed. Reboot persistence, live mining and hardware performance are outside the hosted setup test.

Job 3024 failed with initial control HTTP 000 after the three-second sleep; its raw log and control artifact show both units active but do not prove listener readiness or the underlying startup cause. The delayed-listener regression failed against the pre-change control harness at 4cadfaa (exit 1, delayed control listener was not awaited before POST) and passes after reusing the readiness probe. The unavailable-listener case requires no POST and config restoration. No authenticated write is retried. The completed 3025 log records access-token-readiness: service=control stage=setup elapsed_s=8 before the first POST was accepted and the control round trip completed. This confirms readiness can exceed the old three-second sleep. Job 3019's earlier HTTP 000 remains undiagnosed; the passing jobs do not establish the precise startup cause of either earlier failure.

Local checks passed: timeout 120 bash tests/test-refresh-window-control.sh, timeout 120 bash tests/test-access-token-readiness.sh, bash tests/test-sister-reconcile.sh, timeout 120 bash tests/test-appliance-pool-control.sh, shellcheck --severity=warning tests/e2e-pithead-control.sh tests/test-refresh-window-control.sh, pinned shfmt 3.14.1 -i 4 -d on those two files, make lint-md, FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh, and git diff --check. No ceiling raised. Earlier setup-change checks also passed shellcheck/shfmt for tests/test-stock-home.sh and make lint-actions. No local Docker or hardware tests ran.

Verifier and security review are clean for both the full-setup correction and readiness change. Ponytail review retained the required setup regression, CLI uninstall cleanup and existing readiness helper; no additional product mechanism was needed.

bench-ci job 3019@c7c2868a690eab5b28c6024f4ad748f0bfff2134 · rigforge · tier4-e2e

• options: {"phases": ["worker-api", "control"]}
• status: completed · conclusion: failure · exit 1 · 2m 39s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run failure · collect success · restore success · cleanup success
• rows: 5 pass · 1 fail · 0 skip
• failed rows:
  • [control] POST /apply returned HTTP '000' (expected 202) (no issue linked)
    Generated by bench-ci evidence 3019@c7c2868a690eab5b28c6024f4ad748f0bfff2134; re-run it to check.

bench-ci job 3023@2690c10f56ee501ee304471036bacd21205b18db · rigforge · tier4-e2e

• options: {"phases": ["control"]}
• status: completed · conclusion: success · exit 0 · 7m 35s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
• rows: 5 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 3023@2690c10f56ee501ee304471036bacd21205b18db; re-run it to check.

bench-ci job 3024@4cadfaafd97beb7740708067f4edfa0fe329ced1 · rigforge · tier4-e2e

• options: {"phases": ["control"]}
• status: completed · conclusion: failure · exit 1 · 1m 39s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run failure · collect success · restore success · cleanup success
• rows: 0 pass · 1 fail · 0 skip
• failed rows:
  • [control] POST /apply returned HTTP '000' (expected 202) (no issue linked)
    Generated by bench-ci evidence 3024@4cadfaafd97beb7740708067f4edfa0fe329ced1; re-run it to check.

bench-ci job 3025@b50bc7eddab80ceb72cf0ecf350e77848c74643c · rigforge · tier4-e2e

• options: {"phases": ["control"]}
• status: completed · conclusion: success · exit 0 · 7m 57s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
• rows: 5 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 3025@b50bc7eddab80ceb72cf0ecf350e77848c74643c; re-run it to check.

[pithead-reviewer]

FAIL

What I investigated

Reviewed #595 at 2690c10 against #362 and the supplied pithead#3400 requirements. The apply fix has passing regression and integration evidence, but the required regression through documented stock-host setup is missing.

• gh issue view 362: read the owner’s reports and latest ruling requiring documented setup as a normal user on stock Ubuntu; supplied pithead#3400 context also read.
• gh pr view 595: trusted App author, develop base, reviewed head confirmed; no previous reviews or PR comments.
• git diff origin/develop...HEAD: reviewed all 11 changed files and connected callers. No actionable security, comment-discipline or over-engineering findings.
• gh pr checks 595 --json name,state,bucket,link: nine SUCCESS checks, zero non-pass checks; confirmed again before verdict.
• gh run view 40388132372 --job 121801725361 --log: reopened historical false-success RED and current GREEN assertions. Both exercise apply, neither setup.
• gh run view 40388132372 --job 121801725536 --log: raw evidence confirms 95.14% total coverage against an 80% floor and 100% patch coverage.
• bench-ci evidence 3019@c7c2868a690eab5b28c6024f4ad748f0bfff2134: failure, five pass, one fail, zero skip. Pasted block matches live evidence.
• bench-ci log 3019@c7c2868a690eab5b28c6024f4ad748f0bfff2134: worker-api passes; initial control POST returns HTTP 000. Startup cause remains unestablished.
• bench-ci evidence 3023@2690c10f56ee501ee304471036bacd21205b18db: success, five pass, zero fail, zero skip; baseline verified and rig restored. Pasted block matches live evidence.
• bench-ci log 3023@2690c10f56ee501ee304471036bacd21205b18db: reopened head deployment, authenticated control apply, terminal applied status, persisted config and fresh sister-feed assertions.
• bash tests/test-sister-reconcile.sh: exit 0, PASS.
• Reverted production-only hunks in a detached scratch worktree and ran the sister regression in apply mode: exit 1, 'apply returned success despite a failed reconcile'.
• bash tests/test-refresh-window-control.sh: exit 0, PASS.
• bash tests/test-appliance-pool-control.sh: exit 0, PASS.
• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: exit 0; ceilings remain monotonic.
• git diff --check origin/develop...HEAD: exit 0.
• make lint-topology: exit 0; self-tests and repository scan pass.
• make lint-md: exit 0; zero issues.
• shellcheck --severity=warning on all changed shell files: exit 0.
• Runner topology detector over the PR body, PR comments/reviews, issue e2e-real: pre-flight the checkout path's traversability for the DynamicUser receiver #362 comments and drafted verdict: exit 0, clean.
• git status --short: clean. No local Docker or hardware tests ran. Reviewer GitHub access is read-only; the runner must publish this verdict.

Reviewed head: 2690c10f56ee501ee304471036bacd21205b18db.

What are the gaps

• tests/test-stock-home.sh:45: The owner’s latest e2e-real: pre-flight the checkout path's traversability for the DynamicUser receiver #362 ruling requires a regression running documented setup on stock Ubuntu as a normal user beneath a 0750 home. This fixture creates an empty build directory and fake stopped miner, then invokes only apply; the PR explicitly excludes full stock-host setup. Job 3023 exercises an existing installation. Add the real setup path to a registered hosted/RC test, require authenticated port 8081 without relaxing home permissions, and quote historical RED and current-head GREEN runs.

[pithead-reviewer]

PASS

What I investigated

Re-review of PR #595 (closes #362) at head b50bc7e. The earlier reviewed head 2690c10 is in this clone. Since then the worker added two test-only commits, 4cadfaa and b50bc7e. They touch only tests/, tests/README.md and ci.yml. Product code (rigforge.sh, systemd templates, docs/operations.md) is byte-identical to c7c2868. The round-1 finding is fixed. tests/test-stock-home.sh now runs the documented sudo ./rigforge.sh setup as a normal user from a checkout under a 0750 HOME, on a fresh Ubuntu 24.04 hosted runner, and the job is registered in ci.yml. The test checks that setup compiled a real XMRig binary and installed its unit. It requires an active rigforge-api service and an authenticated 200 on port 8081 before any apply, and HOME stays 0750 throughout. At the RC4 found SHA b2d4c3d the same setup reproduces the defect: setup completes, port 8081 stays down, and the journal shows the API entrypoint (api-server.py) denied. At head it passes. Bench-ci job 3025 on the head passes the control phase, and the new readiness regression fails when the fix is reverted.

• Round-1 finding (tests/test-stock-home.sh:45, apply-only fixture): FIXED. The diff 2690c10..b50bc7e removes the fake xmrig unit and the empty build directory. The test now asserts the build directory is absent before setup, runs real setup, then requires xmrig --version to contain XMRig and the installed unit's ExecStart to point at the built binary. Its setup-listener and authenticated-setup-health stages run before any apply, and stat on HOME must return 750.

• git cat-file -e 2690c10f… succeeded. git log --no-merges 2690c10..b50bc7e lists 4cadfaa and b50bc7e. git diff --stat c7c2868..HEAD -- . ':!tests' ':!.github' printed nothing, so product code is unchanged since c7c2868.

• gh pr checks 595: all 10 checks SUCCESS, none skipped. Run 40400405850 reports headSha b50bc7e and conclusion success.

• Job log 121844421384 (historical, TEST_REVISION b2d4c3d…) printed at 21:04:41: stock-home setup historical RED: real setup completed but the sister entrypoint was denied and port 8081 unavailable. That branch asserts http_code=000 and that the journal contains both 'Permission denied' and 'api-server.py'.

• Job log 121844421165 (current, TEST_REVISION HEAD on merge ref ae6476c of b50bc7e) printed stock-home setup GREEN: real setup built XMRig, installed its unit, and served authenticated port 8081 beneath a 0750 HOME, then stock-home apply GREEN: failed reconcile rejected, service recovered, HOME remains 0750.

• bench-ci evidence 3025@b50bc7e… (rc 0) matches the PR's pasted block exactly: tier4-e2e, phases [control], completed, success, exit 0, 5 pass, 0 fail, 0 skip, rig restored yes. The job log checks out b50bc7e on the rig and records access-token-readiness: service=control stage=setup elapsed_s=8 before ✓ POST /apply accepted. It ends with 5 ✓ rows and exit 0.

• Jobs 3019, 3023 and 3024 are on older SHAs and the body labels them as history. 3025 covers the head for control. The worker-api rows in 3019 at c7c2868 passed on product code identical to the head.

• Scratch worktree at b50bc7e: bash tests/test-refresh-window-control.sh printed thermal refresh-window regressions: PASS (rc 0). After checking out tests/e2e-pithead-control.sh from 4cadfaa (readiness fix reverted) the same run printed delayed control listener was not awaited before POST and POST /apply returned HTTP '000' (rc 1).

• Registration: tests/run.sh:7379 sources test-api-refresh-liveness.sh, which runs test-refresh-window-control.sh (lines 257-258), and tests/run.sh:7150 runs test-sister-reconcile.sh. The CI Test suite log 121844421428 shows ✓ sister API reconciliation reports failures (#362), ✓ thermal refresh-window regression suite (#546) and 2464 passed, 0 failed. test-stock-home.sh is registered at .github/workflows/ci.yml:155.

• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: file budget OK (rc 0).

• Comment discipline: git diff 2690c10..HEAD | grep -E '^\+\s*#' found no added comment lines (rc 1). The delta deletes the old header comment in e2e-pithead-control.sh.

• Topology: detect-topology.py over the PR body, PR comments, review comments, reviews and issue e2e-real: pre-flight the checkout path's traversability for the DynamicUser receiver #362 comments returned rc 0 with output clean.

• Ponytail pass, done inline over the 132-line delta: the control harness reuses the existing _access_tokens_ready helper instead of adding a new wait. The matrix and curl-stub additions are what the regressions need. Nothing to cut.

• Security: the delta is test-only. The fixture token appears only inside a disposable hosted-runner config, and the test adds no service, listener or sudo grant.

• Acceptance (e2e-real: pre-flight the checkout path's traversability for the DynamicUser receiver #362 owner ruling 2026-10-12, pithead#3400 case 3): met. Apply fails loudly when the sister API is inactive (test-sister-reconcile.sh plus the missing-entrypoint stage). Normal-user setup under a 0750 home leaves rigforge-api active with 8081 listening. The test is red at the found SHA b2d4c3d and green at head, and the placement preflight is kept.

• tests/test-stock-home.sh:69, tests/test-stock-home.sh:81: ! systemctl is-active --quiet … is inert under set -Eeuo pipefail: bash exempts negated commands from errexit. Checked locally: bash -c 'set -Eeuo pipefail; ! true; echo still running' prints still running. So the xmrig-stopped check and the historical branch's rigforge-api-inactive check never fail. The RED still holds on its other assertions (http_code=000 plus Permission denied on api-server.py in the journal). Fix: if systemctl is-active --quiet X; then exit 1; fi.

Reviewed head: b50bc7eddab80ceb72cf0ecf350e77848c74643c.

[pithead-reviewer]

Owner brief for b50bc7eddab8

What it does. The status API and control receiver can start from a checkout beneath a private home directory without loosening its permissions. Setup and apply report status API startup failures instead of silently succeeding; apply still completes its remaining updates.

Why. Closes #362 in v1.18.1. The services’ restricted accounts could not read their Python programs beneath a private home directory, leaving receivers unavailable while setup or apply could report success.

Risk. Affects Linux receiver startup and setup/apply results. An unavailable status API now causes failure after a bounded wait, and apply may already have made other changes when it reports failure. No irreversible change found.

Proof. CI: 11 success. Reviewer: PASS at b50bc7e by pithead-reviewer, round 2 (recorded 2026-10-26T21:14:12Z). Bench: job 3019: failure on c7c2868, not the head; job 3023: success on 2690c10, not the head; job 3024: failure on 4cadfaa, not the head; job 3025: success on the head.

What to check.

• systemd/rigforge-api.service.template:6 and systemd/rigforge-control.service.template:6: systemd supplies readable copies of the programs while both services retain restricted accounts.
• rigforge.sh:1415 and rigforge.sh:4229: the listener wait is bounded, and API failure reaches the caller after remaining updates complete.
• tests/test-stock-home.sh:59: the regression uses ordinary setup and requires authenticated API access before apply while home permissions remain private.
• tests/test-stock-home.sh:69 and :81: the reviewer noted that the negated service-state checks do not enforce their assertions.

Verdict. merge