Skip to content

GH-43: react to request-level 5xx in the health model - #44

Merged
blainemotsinger merged 7 commits into
mainfrom
GH-43
Oct 4, 2026
Merged

blainemotsinger merged 7 commits into
mainfrom
GH-43

Conversation

@blainemotsinger

Copy link
Copy Markdown
Member

Summary

Closes #43. The health monitor's probe is liveness-only (vLLM /health stays 200 while saturated), so overload was invisible to load balancing. This branch feeds each proxied attempt's HTTP status back into the monitor and elevates overloaded nodes to the degraded fallback pool.

Fixes follow the approved design (tmp scratch, not committed):

  • Hit rule — only status >= 500 is a request-level hit; any other status (incl. 4xx/429) clears the streak.
  • Elevation cap — request elevation goes to degraded at request_overload_threshold (default 3); never dead (death stays probe-owned). The GetNodeStates merge lifts only probe-healthy nodes.
  • Recovery — read-evaluated time decay, 10s after the last hit; the stored streak is never mutated by reads (req_streak on /health reflects it).
  • One new config key — [gateway] request_overload_threshold (default 3, 0 = default, negative = invalid).
  • No new Prometheus metrics; balancer, connection-failure offline path, and Recheck sites untouched.
  • Three mutually exclusive report sites in the proxy paths (streaming, committed, buffered — one report per status-bearing attempt).

Changes

  • config: new key + accessor/validation + tests (7 commits: 19b1353..17e55e6)
  • health: ReportOutcome, streak fields, read-time merge elevation, reload carryover, TTL decay
  • api: outcome reporting from the proxy paths, req_streak in /health
  • docs: README + api.md
  • tests: unit (config, elevation math, decay, carryover, concurrency) + integration (round-robin elevation, streaming + committed sites) + 8/9 mutation gates killed

Verification

gofmt / vet / go test ./... / go test -race (health + api) all green at head.

Known gap (follow-up issue filed)

Dropping the streaming guard (gate.err == nil) is not mutation-gate covered; a truncated-stream 200 would currently clear the streak ("truncated-stream 200 must not clear ReqStreak").

@blainemotsinger
blainemotsinger merged commit c884164 into main Oct 4, 2026
4 checks passed
@blainemotsinger
blainemotsinger deleted the GH-43 branch October 4, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overload sensing: react to request-level 5xx in the health model

1 participant