[iamwavecut]

Problem

From 2026-11-05 15:26 to 2026-11-06 21:58 UTC the GPU2 backend (ninfer-3090) was not running. Discovery failed every job with result.error = {code: "UPSTREAM_ERROR", message: "[Errno -3] Temporary failure in name resolution"}. The bot read only the top-level job.error, so:

• Every failure became the generic dialog job failed. It matched no retry rule, so the walker never charged the breaker, never emitted circuit_open_exhaustion, and the owner's routing incident report stayed silent for 30 h. Before the reboot, while the same backend returned connection refused, that report had fired.
• Each memory run spent an attempt per failure (backoff 1/4/9/16 min), so a run was retired after about 30 min. 7,897 runs (842k messages) were retired, and the Analytics job pipeline showed ~29k failed memory_consolidation jobs per day.

Change

• Discovery result.error is decoded (DiscoveryJobResult.error, DiscoveryJobStatus.error_code). A failed job now reports CODE: message. The codes that mean the backend was never reached (UPSTREAM_ERROR, SERVICE_NOT_FOUND, SERVICE_DISABLED, ENDPOINT_NOT_FOUND) produce the new AifarmClientError::BackendUnavailable, which retryable_reason classifies as ProviderUnavailable, as it already does for Transport. UPSTREAM_TIMEOUT and unknown codes stay Upstream.
• Memory runs no longer spend attempts on an LLM outage. When extraction fails because no backend could take the call (ProviderUnavailable/CapacityUnavailable, or routing NoCandidates/Exhausted/RouteUnavailable/CapacityWaitTimeout), the run is released, as the embedder gate already does. Other extractor errors still fail the run.
• A deferred memory pipeline collapses to one delayed probe. A deferred tick queues a followup only when no other memory job is pending, and that job becomes available after MEMORY_CONSOLIDATION_BACKEND_RETRY_INTERVAL (15 s, renamed from the embedder-only constant). The first productive probe ramps the pipeline back up through the existing amplification.
• Owner signal. No new channel: with the typed error the walker charges the breaker, and the existing routing admin report delivers circuit_open_exhaustion. A walker test pins this for BackendUnavailable and shows that the old generic error never reached the report.

Verification

• cargo fmt --all
• cargo clippy --workspace --all-targets -- -D warnings: passed
• cargo test -p openplotva-llm -p openplotva-app -p openplotva-taskman: 1,460 app + 225 llm + 68 taskman tests pass
• New tests:
  • aifarm: a real Discovery failed-job JSON decodes to UPSTREAM_ERROR: …; the four codes map to BackendUnavailable; timeout, internal and bare failures stay Upstream/dialog job failed.
  • retry: BackendUnavailable is ProviderUnavailable although its text matches no phrase rule.
  • memory_runtime: classification table (unreachable, transport, 503 loading, capacity and routing errors release; timeout, generic, response and deadline errors and store failures do not); an unreachable backend releases the run without failing it; a timeout still spends the attempt; three deferring workers leave exactly one delayed probe job; the embedder deferral's followup is not dequeued before the retry interval.
  • routed_attempts: six BackendUnavailable failures open the circuit and queue an aggregated admin report; six generic dialog job failed failures do not.

Operations already done (not part of this diff)

• ninfer-3090 was recreated on the farm at 21:58 UTC; memory extraction on GPU2 runs without errors.
• The 7,897 outage runs got attempts = 0 (status kept failed, so they stay outside MEMORY_MAX_QUEUED_RUNS) and are draining at ~14 runs/min.
• iamwavecut/aifarm-discovery#7: net.ipv4.ip_nonlocal_bind=1 and a container watchdog that recreates compose services Docker failed to start. Installed on the farm.

Compatibility

No schema, prompt, route or Redis change. Operator-visible error texts change from dialog job failed to Discovery's CODE: message when Discovery provides one. Unreachable-backend failures become retryable for every routed workflow, so the walker falls back and charges breakers on them, as it already did for connection refused.

[github-actions]

	Messages
📖	Danger checked 5 changed file(s).

Generated by 🚫 dangerJS against 0aed700