Skip to content

GH-19: proxy connection-level retry with failover - #23

Merged
blainemotsinger merged 8 commits into
mainfrom
GH-19
Sep 20, 2026
Merged

blainemotsinger merged 8 commits into
mainfrom
GH-19

Conversation

@blainemotsinger

Copy link
Copy Markdown
Member

What

Connection-level retry for the gateway proxy.

  • Buffered attempts: each node attempt buffers the full response in memory; the client gets exactly one committed write on success. Fixes fabricated 200s on truncation and committed 502s.
  • Tiered retry: dial-phase failures and timeouts (incl. response-header timeout, documented duplicate-generation risk) retry on other pool nodes; HTTP statuses from vLLM, write-phase errors, truncated responses, and client aborts never retry.
  • Negative feedback: per-IP 10s offline skip-set + Monitor.Recheck on failure transitions; write-phase failures also mark offline.
  • Failover correctness: per-iteration candidates exclude tried/offline nodes so Degraded survivors are reachable during the monitor's convergence window.
  • Metrics: gateway_requests_total once per request (final status); new gateway_proxy_attempts_total{model,status} per attempt.
  • Config: Validate() rejects duplicate node IPs (node identity is IP-keyed).
  • Docs: README limitations, api.md 502/503 semantics, architecture retry section.

Verification

  • go test ./... -race -count=1 pass; go vet, go build, gofmt clean.
  • Per-task TDD with RED→GREEN evidence; task-scoped and whole-branch adversarial reviews (6 commits + 2 fix commits, all review clean).

Closes #19

@blainemotsinger
blainemotsinger merged commit 2ece883 into main Sep 20, 2026
4 checks passed
@blainemotsinger
blainemotsinger deleted the GH-19 branch September 20, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proxy retry/fallback

1 participant