#45 fix: liveness in-flight means the post hasn't returned, not that the thread is alive

[gregoryfoster]

Refs #45 (closed after deploy and the live check).

Cause. A check-in thread hands its result to the loop (call_soon_threadsafe) and then exits. The tick could finish, and a following tick that awaits nothing (stale/skipped) let the next one run, while that thread was still alive: is_alive() → spurious skip. Red on main three times in two days.

Fix (src/processor/liveness.py): an explicit _in_flight marker replaces _thread.is_alive().

• set before start(); cleared by the thread once post returns, before call_soon_threadsafe;
• never cleared by the timeout path: a post cut off by asyncio.timeout stays in flight until it returns, so threads still never pile up (Liveness check-in for processor run (co-status monitor co-processor-live) #39 trap 1);
• a failed start() clears it, or every later tick would skip.

Tests (tests/test_liveness.py):

• test_a_finished_check_in_whose_thread_lingers_does_not_skip_the_next_tick: holds each check-in thread after its target returns. Deterministic, no sleeps. Red before the fix (1 == 2, the skip warning), green after.
• test_a_thread_that_fails_to_start_does_not_wedge_the_heartbeat: guards the marker (mutation-checked: fails without the except clear).
• test_a_hung_status_is_cut_off_and_the_next_tick_is_skipped (existing) still passes: a post past the timeout still skips the next tick.
• tests/test_consumer.py integration test now asserts beat._in_flight.

Verification.

• uv run pytest: 589 passed, 6 skipped (unreadable-secret denials, expected in a worktree).
• uv run pytest -m integration: 56 passed.
• tests/test_liveness.py ×100 under 4 CPU burners on 2 vCPU: 100/100.

After merge: scripts/deploy.sh (touches src/). Live check: starting has liveness: on …, then liveness check-in with no WARNING over the next ticks, co-processor-live stays ok.

🤖 Generated with Claude Code