Skip to content

feat(gateway): adapt upstream response modes through bounded probes - #401

Merged
fylorn merged 6 commits into
mainfrom
codex/adaptive-stream-mode
Nov 12, 2026
Merged

fylorn merged 6 commits into
mainfrom
codex/adaptive-stream-mode

Conversation

@fylorn

@fylorn fylorn commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Generation requests follow the client's response mode by default. An eligible HTTP 400/422/501 JSON API rejection can trigger one controlled trial of the opposite mode on the same upstream. This works with opaque or localized rejections without requiring a specific mode-error code or English phrase. The gateway converts a successful answer back to the client's protocol and framing.

Known backend requirements take priority: ChatGPT account backends retain their existing stream-only handling. Known history, reasoning-effort and cache-mark repairs run before the generic probe, sharing one automatic repair budget per logical request and respecting its wait deadline. Recognizable account/capacity errors, retry hints and explicit non-mode parameter validations are excluded. An unrelated opaque request error can cost one extra rejected attempt.

If the probe is also rejected, disconnects or fails before content, retain the original error with its own headers, conversion context and redaction ledger. Record both actual attempt outcomes. Original server errors still permit failover to another provider; later candidates start with the original client request. HTTP-success JSON error bodies in trial responses also preserve the original error. Partially delivered answers are never replayed.

A complete, clean answer teaches a working mode preference for a fixed 30-minute window. Cached uses do not extend expiry: the next request after expiry tries the client's mode again. Entries are isolated by account fingerprint, model, API/path and request options, bounded at 1,024, cleared by configuration reload and invalidated on an eligible rejection. No prompts or raw credentials are retained in this cache.

Validation before integration with the latest main

  • CARGO_INCREMENTAL=0 cargo nextest run -p tw-gateway --profile ci: 1,559 passed, 2 skipped.
  • CARGO_INCREMENTAL=0 cargo clippy -p tw-gateway --all-targets -- -D warnings: passed.
  • cargo fmt --all -- --check, git diff --check: passed.
  • python3 scripts/release_notes_test.py: 17 passed.
  • Fifteen HTTP integration tests cover both directions and actual client framing across Chat, Anthropic, Responses, Gemini and Bedrock; tool arguments and usage; opaque/localized rejections; failed probes and original-error preservation; attempt outcomes, auth/limit state and failover; incomplete/embedded-error responses; learning and reload. Two unit tests cover probe eligibility and fixed cache expiry/isolation/bounds.
  • Full-workspace CI passed on the final commit across macOS, Linux and Windows: Clippy, all workspace tests and platform binary checks. Linux reports 4,427 passed, 9 skipped. macOS/Linux run the data-plane smoke suite; Windows verifies startup, control-plane handshake and shutdown.

Retries use the normal authorization, plugins, redaction, request recording and usage accounting paths and reuse the upstream slot. Existing gateway regressions for schema repairs, client-specified budgets/cache points, redaction and tool filtering retain their assertions.

Limits

Whole-to-stream conversion still waits for the upstream's complete answer. Native requests requiring multiple candidates, log probabilities or non-text output are excluded because the current response IR cannot preserve them. WebSocket requests are excluded.

Related discussion: ThinkWatchProject/ThinkWatch-Lite#407 (comment). A Rate limit exceeded response does not establish a mode restriction and does not initiate a probe. This PR does not establish the root cause of that report; validation uses simulated upstream HTTP responses, not a reproduction against the reporter's upstream.

Release preparation

Prepares Core 0.96.0 and integrates the latest main, retaining automatic discovery and manual model-capability overrides from Core 0.95.0. All 22 workspace package versions and the lockfile are synchronized. The integrated head passes locked offline workspace metadata validation, Rust formatting, the 17 release-note tests, and diff checks locally. The integrated release head passed full-workspace CI: macOS, Linux and Windows Clippy/tests and real binary checks, TypeScript bindings, and enterprise compatibility. Linux reports 4,438 passed / 9 skipped; Windows reports 4,420 passed / 8 skipped. The dependency audit also passed. Lite 2026.11.14 will pin the published v0.96.0 tag.

Retry one explicitly rejected response mode using the alternative and
convert the result to the client protocol. Learn only complete successful
answers, sharing the existing automatic repair budget.
Allow one controlled opposite-mode trial after eligible opaque API request
rejections, after specific schema repairs. Exclude account/capacity errors,
retry hints and explicit non-mode validations.

Keep the original error and response context if the probe also fails before
content; record its actual outcome without repeating the failed trial.
Preserve normal failover for original server errors and handle embedded JSON
errors in HTTP-success trial responses.

Learn only complete working answers for a fixed 30-minute window, then
retry the client mode. Extend HTTP regressions for custom/localized errors,
failed probes, attempt records, account states and candidate isolation.
@fylorn fylorn changed the title feat(gateway): negotiate upstream streaming capabilities feat(gateway): adapt upstream response modes through bounded probes Nov 12, 2026
@fylorn
fylorn marked this pull request as ready for review November 12, 2026 15:40
@fylorn
fylorn merged commit a49d0a8 into main Nov 12, 2026
7 checks passed
@fylorn
fylorn deleted the codex/adaptive-stream-mode branch November 12, 2026 16:06
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 12, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant