Repository navigation
feat(gateway): adapt upstream response modes through bounded probes - #401
Merged
Merged
Conversation
Retry one explicitly rejected response mode using the alternative and convert the result to the client protocol. Learn only complete successful answers, sharing the existing automatic repair budget.
Allow one controlled opposite-mode trial after eligible opaque API request rejections, after specific schema repairs. Exclude account/capacity errors, retry hints and explicit non-mode validations. Keep the original error and response context if the probe also fails before content; record its actual outcome without repeating the failed trial. Preserve normal failover for original server errors and handle embedded JSON errors in HTTP-success trial responses. Learn only complete working answers for a fixed 30-minute window, then retry the client mode. Extend HTTP regressions for custom/localized errors, failed probes, attempt records, account states and candidate isolation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Generation requests follow the client's response mode by default. An eligible HTTP 400/422/501 JSON API rejection can trigger one controlled trial of the opposite mode on the same upstream. This works with opaque or localized rejections without requiring a specific mode-error code or English phrase. The gateway converts a successful answer back to the client's protocol and framing.
Known backend requirements take priority: ChatGPT account backends retain their existing stream-only handling. Known history, reasoning-effort and cache-mark repairs run before the generic probe, sharing one automatic repair budget per logical request and respecting its wait deadline. Recognizable account/capacity errors, retry hints and explicit non-mode parameter validations are excluded. An unrelated opaque request error can cost one extra rejected attempt.
If the probe is also rejected, disconnects or fails before content, retain the original error with its own headers, conversion context and redaction ledger. Record both actual attempt outcomes. Original server errors still permit failover to another provider; later candidates start with the original client request. HTTP-success JSON error bodies in trial responses also preserve the original error. Partially delivered answers are never replayed.
A complete, clean answer teaches a working mode preference for a fixed 30-minute window. Cached uses do not extend expiry: the next request after expiry tries the client's mode again. Entries are isolated by account fingerprint, model, API/path and request options, bounded at 1,024, cleared by configuration reload and invalidated on an eligible rejection. No prompts or raw credentials are retained in this cache.
Validation before integration with the latest main
CARGO_INCREMENTAL=0 cargo nextest run -p tw-gateway --profile ci: 1,559 passed, 2 skipped.CARGO_INCREMENTAL=0 cargo clippy -p tw-gateway --all-targets -- -D warnings: passed.cargo fmt --all -- --check,git diff --check: passed.python3 scripts/release_notes_test.py: 17 passed.Retries use the normal authorization, plugins, redaction, request recording and usage accounting paths and reuse the upstream slot. Existing gateway regressions for schema repairs, client-specified budgets/cache points, redaction and tool filtering retain their assertions.
Limits
Whole-to-stream conversion still waits for the upstream's complete answer. Native requests requiring multiple candidates, log probabilities or non-text output are excluded because the current response IR cannot preserve them. WebSocket requests are excluded.
Related discussion: ThinkWatchProject/ThinkWatch-Lite#407 (comment). A
Rate limit exceededresponse does not establish a mode restriction and does not initiate a probe. This PR does not establish the root cause of that report; validation uses simulated upstream HTTP responses, not a reproduction against the reporter's upstream.Release preparation
Prepares Core 0.96.0 and integrates the latest main, retaining automatic discovery and manual model-capability overrides from Core 0.95.0. All 22 workspace package versions and the lockfile are synchronized. The integrated head passes locked offline workspace metadata validation, Rust formatting, the 17 release-note tests, and diff checks locally. The integrated release head passed full-workspace CI: macOS, Linux and Windows Clippy/tests and real binary checks, TypeScript bindings, and enterprise compatibility. Linux reports 4,438 passed / 9 skipped; Windows reports 4,420 passed / 8 skipped. The dependency audit also passed. Lite 2026.11.14 will pin the published v0.96.0 tag.