GH-31: per-node model awareness

[blainemotsinger]

Per-node model awareness (GH-31)

Closes #31

Summary

The gateway now discovers each node's served models via GET /v1/models and excludes nodes that cannot serve the requested model, without coupling discovery failures to health.

What changed

• Model discovery state machine (internal/health/): per-node ModelDiscovery with states untried / known / known_empty / degraded / expired. Exact string equality on data[].id — no aliases.
• Isolated discovery pipeline: discoverAll is a non-blocking spawn with its own modelsChecking inflight map. Listing failures never touch Status, ConsecutiveFailures, or the health fail threshold. A hung /v1/models cannot delay health checks (proven by test).
• State semantics:
  • untried (zero successful fetches, Fails < N) is a filter identity — nodes pass. No Start() ⇒ stay untried.
  • Two consecutive successful data: [] responses enter known_empty (excluded from all model routing).
  • Fetch failure keeps Names (last-known-good). Never cleared on error.
  • After health_fail_threshold consecutive listing failures → expired (excluded until listing recovers). No wall-clock stale deadline.
  • Bodies without a data field are parse errors, not empty successes — an error body on HTTP 200 cannot poison known_empty.
• Request-path filter (internal/api/): after filterByRing, nodes are filtered by Allows(modelName) on the injected model name. Per-node exclusion reasons (missing/empty/expired) are counted. Empty pool still returns 503 no healthy node available (unchanged, alert-safe).
• /health is additive only: status/nodes/healthy unchanged. New models_ok (count of known+known_empty nodes) and details[].models{state,names,age_seconds,fails,last_error}.
• 3 new metrics: gateway_model_filter_exclusions_total{model,reason}, gateway_node_models_ok{node}, gateway_model_discovery_total{node,result}. All existing metric names/labels and log messages are immutable.
• New transition-only logs: models discovered, node reports no models, model discovery failed, model discovery expired, model discovery recovered, node model list changed, configured model not served by node. Exactly one message per state transition.
• SIGHUP reload carries Models state (incl. Fails/Empties/LastError) across NewMonitorWithState; GetNodeStates deep-copies Models.Names.

Behavior change (documented)

Standalone/ad-hoc request models are now filtered the same way as configured models — a model missing from every node's known list returns 503. models.default and every [[models.available]].name must equal the node's served model id exactly. Discovery failures never mark a node unhealthy. See the README Limitations bullet.

Soft-reject policy

warnConfigModelsMissing warns configured model not served by node once per (node, model) per list change when a configured model that should be on the node is absent from its list. It never rejects config. models.default is checked only when it is absent from [[models.available]] entirely.

Verification

• Full suite go test ./... green; -race on api/health/metrics; GOTOOLCHAIN=go1.22.12 (CI pin); -count=5 stability on api/health.
• make test vet fmt clean.
• 8/8 mutation gates killed and reverted (filter, state machine, deep-copy, reload carryover, untried identity, 503 path, isolation).
• Re-review of fix wave: Ready to merge.

Follow-ups (not blocking)

• GetNodeStates aliases the GPUs backing array across the lock boundary #38 — GetNodeStates aliases the GPUs backing array (pre-existing)
• Bound /v1/models and metrics-scrape response body reads #39 — bound /v1/models and metrics-scrape response body reads (hardening)
• GH-31 follow-up: model-discovery test hardening #40 — model-discovery test hardening (coverage gaps recorded from reviews)

Review notes

Two plan-mandated deviations were user-approved during execution: parseModelList rejects bodies without a data field (error bodies cannot feed known_empty), and the transition-log switch emits exactly one message per transition per spec §6.3. filterByModel in snapshot.go is a tested helper unused in production (handler uses an inline loop for per-node reason labels) — pre-accepted.