Skip to content

Fall back locally after mediator HTTP errors - #1493

Merged
prk-Jr merged 7 commits into
mainfrom
fix/mediator-http-fallback
Nov 12, 2026
Merged

prk-Jr merged 7 commits into
mainfrom
fix/mediator-http-fallback

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Preserve collected provider bids when adserver_mock returns a non-2xx HTTP status. The shared parser now returns an error, allowing the existing warning and local-ranking fallback to run with floor enforcement.
  • Attribute local winning-bid telemetry to the selected bid using bid identity and price, including mediator fallback. Preserve mediated provider association when mediation changes bid IDs or supplies decoded prices; failed mediation still omits its mediator row.
  • Document mediator failure handling in the auction and integration guides.

Changes

File Change
crates/trusted-server-core/src/integrations/adserver_mock.rs Return a typed error containing the HTTP status for non-success responses; update parser error documentation.
crates/trusted-server-core/src/auction/orchestrator.rs Clarify response-processing warnings; test fallback ranking, floors, per-bid telemetry and successful/no-bid mediation in production and harness paths.
crates/trusted-server-core/src/auction/telemetry.rs Match selected local bids by ID and price; cover missing/reused IDs, cross-provider collisions, ties and rewritten mediator IDs.
docs/guide/auction-orchestration.md Describe local fallback, floors, warnings, and telemetry behavior.
docs/guide/integrations/adserver_mock.md Explain mediator failure fallback and valid no-bid decisions; use “logs” to include debug-level deadline handling.

Closes

Closes #1203

Spec

Not a feature. This bug fix restores the documented mediation-failure fallback contract.

Test plan

  • cargo test-fastly && cargo test-axum
  • cargo clippy-fastly && cargo clippy-axum
  • cargo fmt --all -- --check
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run
  • JS format: cd crates/trusted-server-js/lib && npm run format
  • Docs format: cd docs && npm run format
  • WASM release build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1 (not run)
  • Manual testing via fastly compute serve (not run)
  • Full native core suite: 3,344 passed, two ignored.
  • Focused mediator and telemetry suite: 61 passed. Strengthened production/harness fallback tests and the new local telemetry regression failed with the previous matcher and passed after correction.
  • Remaining required local gates: reusable-sandbox Fastly, Cloudflare and Spin tests; all eight Clippy aliases; parity; native build-digest tests/lint; URL-host lint; JS build; Markdown formatting; wasm-clock, documentation-proposal and CI Node-pin checks.
  • Actual Fastly deadline fixtures and Cloudflare/Spin deadline fixture builds, lints and runtime checks.
  • Actual Axum server through normal CLI config validation/local push and POST /auction, with controlled HTTPS bidder and mediator endpoints.

The initial HTTP-status fix was verified through an actual Axum server: the original server lost a collected $2.00 provider bid after mediator 503; the fix preserved it while retaining successful mediation and valid no-bid decisions.

The final telemetry correction was verified through the actual Fastly application under Viceroy using normal strict config validation/local push, controlled HTTPS bidder/mediator/ingestion endpoints, and captured Tinybird NDJSON. Before this correction, the server selected a $2.00 bid but marked an earlier $1.50 bid as winning. The corrected artifact emitted $1.50: is_win=0 and $2.00: is_win=1 for repeated 503s, malformed-response fallback, and no-mediator execution. Valid no-bid and existing mediated association behavior remained intact. This correction targets local winner matching; successful mediation's documented duplicate-identity ambiguity remains unchanged.

Local TLS mappings used task-owned CA certificates through the maintained Viceroy per-backend configuration pattern; TLS verification remained enabled. All owned runtime processes stopped and TLS private keys were removed. Floors were verified in core tests because /auction has no floor input.
Local checks used Rust 1.98.1, Node 24.20.0, Viceroy 0.21.0 and Spin 4.1.0. Checks requiring macOS certificate or loopback access passed outside the restricted sandbox. Hosted CI and deployed runtime behavior are separate from these local results.

Checklist

  • Feature spec requirement: not applicable to this bug fix
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code
  • Uses log macros
  • New code has regression coverage
  • No secrets or credentials committed

@prk-Jr prk-Jr self-assigned this Nov 4, 2026
Match local winning bids by instance ID and price so an earlier bid from
the same bidder cannot claim the win. Preserve mediated association when
mediation rewrites IDs or supplies decoded prices.

Cover missing and reused IDs, provider collisions, ties, and HTTP-error
fallback telemetry. Use level-neutral wording for mediator failure logs.
@prk-Jr
prk-Jr marked this pull request as ready for review November 5, 2026 09:46

@dhruv8sh dhruv8sh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Non-2xx mediator responses now return an error, so they go through the existing local-ranking fallback (with floors) instead of discarding collected provider bids. Local winner telemetry is now matched on bid ID and price. The fix is correct and well tested; only non-blocking notes below.

1 inline comment carries a one-click suggestion (compile-unverified — please run cargo fmt --check after applying).

Non-blocking

  • 🌱 Failed mediation is only visible in logs — inline at adserver_mock.rs:391
  • 📝 Local-win matching assumes winners are verbatim clones — inline at telemetry.rs:761
  • ⛏ Error message wording — inline at adserver_mock.rs:393
  • 👍 Thorough fallback and telemetry tests — inline at orchestrator.rs:7152

CI Status

All 29 checks: PASS.

State on select_winning_bids that winners are verbatim clones of provider
bids, since telemetry attributes local wins by bid_id and price. Restore the
"non-success" wording in the mediator HTTP error so existing log searches
still match.

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

adserver_mock now returns an error for a non-2xx mediator response, so both production collect paths fall back to local ranking with floors instead of dropping the collected bids. Local-ranking telemetry now marks the bid that was actually selected. The fix is correct and well covered: reverting the provider change fails both new fallback tests, and reverting the telemetry clause fails three. One blocking item: the behavior and telemetry shifts need a CHANGELOG entry. The inline comments are non-blocking.

1 of the inline comments below carries a one-click GitHub suggestion. Use Commit suggestion to apply it. The others describe the change in prose because it spans lines outside the diff or asks a question.

Blocking

🔧 wrench

  • CHANGELOG entry missing: see Cross-cutting below

Non-blocking

  • ♻️ Local-win telemetry keys off mediator_response.is_some(): see inline at crates/trusted-server-core/src/auction/telemetry.rs:760
  • 🤔 A mediator HTTP 204 falls back instead of ending as a no-bid: see inline at crates/trusted-server-core/src/integrations/adserver_mock.rs:391
  • ⛏ Mediator failure wording in the guide: see inline at docs/guide/auction-orchestration.md:919
  • 🌱 The orchestrator still treats a BidStatus::Error mediator response as final: see inline at crates/trusted-server-core/src/auction/orchestrator.rs:2656

Cross-cutting / body-level findings

  • 🔧 CHANGELOG entry missing. Both changes in this PR are operator-visible:

    • During a mediator outage, deployments now serve locally ranked bids instead of none.
    • is_win changes for local ranking on every deployment that emits auction telemetry, not only after a mediator fallback. When one bidder code returned several bids for a slot without distinguishing ad IDs, an earlier lower-priced bid (possibly from another provider) was marked as the win at its own price. auction_bid_stats_mv groups wins and winning_cpm_quantiles by provider and seat, so those series shift after upgrading.

    Other recent PRs (#1405, #1406, #1433) update the changelog, and [Unreleased] has a ### Fixed list for bugs like this one. Proposed entries for the top of ### Fixed (prettier-clean when inserted there):

    - A mediator that answers with a non-2xx HTTP status no longer discards the collected provider bids. The orchestrator now logs a warning and ranks those bids locally with floors, as it already did for mediator launch, transport, deadline and parse failures (#1203).
    - Auction telemetry now marks the bid that local ranking selected as the slot's winner. When one bidder code returned several bids for a slot without distinguishing ad IDs, an earlier lower-priced bid, possibly from another provider, was marked as the win at its own price. Tinybird per-provider win counts and winning-CPM quantiles for those slots shift after upgrading. Mediated wins are unchanged.

    Apply manually: CHANGELOG.md is not in this PR's diff, so it can't be a suggestion.

CI Status

  • Analyze (actions): PASS
  • Analyze (javascript-typescript): PASS
  • Analyze (javascript-typescript): PASS
  • Analyze (python): PASS
  • Analyze (rust): PASS
  • browser integration tests: PASS
  • cargo check (cloudflare native + wasm32-unknown-unknown): PASS
  • cargo check/build/test (spin native + wasm32-wasip1): PASS
  • cargo fmt: PASS (required)
  • cargo test: PASS (required)
  • cargo test (axum native): PASS
  • cargo test (cross-adapter parity): PASS
  • cargo test (ts CLI, native) (macos-latest): PASS
  • cargo test (ts CLI, native) (ubuntu-latest): PASS
  • CI Node pin guard: PASS
  • CLAUDE.md symlink guard: PASS
  • CodeQL: PASS
  • docs-proposal scripts: PASS
  • format-docs: PASS (required)
  • format-typescript: PASS (required)
  • HTML EOF runtime (Fastly): PASS
  • integration tests: PASS
  • integration tests (Fastly EC lifecycle): PASS
  • prepare integration artifacts: PASS
  • trace runtime and browser acceptance: PASS
  • ts dev lint domains (changed lines): PASS
  • vitest: PASS
  • worker-deadlines (cloudflare): PASS
  • worker-deadlines (spin): PASS

Comment thread docs/guide/auction-orchestration.md Resolved

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The change restores local ranking after mediator HTTP errors, preserves floor enforcement, and corrects local winner telemetry without changing successful mediation behavior. No verified code findings.

Reviewed fe9a220e6 and checked the update to 734bf8962. The update is a clean merge of main with no conflict-resolution edits. All five reviewed files are unchanged, and the PR patch against its updated base is byte-for-byte identical to the previously reviewed patch.

CI status

28 reported checks pass. worker-deadlines (cloudflare) failed during tooling setup because the GitHub release download for wasm-bindgen returned HTTP 504. The fixture build and Clippy passed before that failure; its runtime test did not execute. This is an external download failure, not evidence of a regression in this patch. That job still needs a successful rerun.

  • browser integration tests: PASS
  • integration tests: PASS
  • trace runtime and browser acceptance: PASS
  • integration tests (Fastly EC lifecycle): PASS
  • CodeQL: PASS
  • prepare integration artifacts: PASS
  • cargo fmt: PASS (required)
  • format-typescript: PASS (required)
  • worker-deadlines (spin): PASS
  • Analyze (rust): PASS
  • vitest: PASS
  • ts dev lint domains (changed lines): PASS
  • HTML EOF runtime (Fastly): PASS
  • cargo test: PASS (required)
  • worker-deadlines (cloudflare): FAIL
  • cargo test (ts CLI, native) (macos-latest): PASS
  • CLAUDE.md symlink guard: PASS
  • Analyze (javascript-typescript): PASS
  • cargo check (cloudflare native + wasm32-unknown-unknown): PASS
  • cargo test (ts CLI, native) (ubuntu-latest): PASS
  • cargo test (cross-adapter parity): PASS
  • cargo check/build/test (spin native + wasm32-wasip1): PASS
  • docs-proposal scripts: PASS
  • format-docs: PASS (required)
  • Analyze (actions): PASS
  • cargo test (axum native): PASS
  • CI Node pin guard: PASS
  • Analyze (python): PASS
  • Analyze (javascript-typescript): PASS

Local git diff --check passed. Passing hosted checks were reused rather than rerun locally; deployed runtime behavior was not independently exercised.

Attribute local telemetry wins to the exact winning provider bid instead of
inferring local ranking from an absent mediator response, so keeping a
failed mediator response cannot reattach the win to an earlier, lower bid.

Treat a mediator HTTP 204 as a final no-bid, matching how bidder providers
handle 204, and fall back to local ranking when a mediator reports an
error-status response on both the production and harness paths.

Clarify the mediator failure wording in the auction guide and add
CHANGELOG entries for the fallback and telemetry changes.
@prk-Jr

prk-Jr commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

@aram356's blocking CHANGELOG item is addressed in 7ff0d45. I added two entries at the top of ### Fixed under [Unreleased], starting from your proposed text:

Prettier with docs/.prettierrc is clean.

Local gates on this commit: cargo fmt --check; the six adapter clippy aliases plus clippy-cli and clippy-codegen; test-fastly and test-fastly-reuse (Viceroy 0.21.0); test-axum, test-cloudflare and test-spin; parity; the URL-host lint; the release WASM build; vitest; and JS and docs format. All passed. Hosted CI for this commit is still pending.

@prk-Jr
prk-Jr requested a review from aram356 November 11, 2026 15:49
@prk-Jr prk-Jr added this to the 202611 milestone Nov 12, 2026
Both the production orchestrator and the planned harness now log a
warning before ranking locally after a mediator reports BidStatus::Error,
matching the other mediator failure paths. Add a regression for each site
and pin exact attribution when a mediator keeps a provider bid's identity.
@prk-Jr
prk-Jr merged commit 81e2350 into main Nov 12, 2026
30 checks passed
@prk-Jr
prk-Jr deleted the fix/mediator-http-fallback branch November 12, 2026 08:23
prk-Jr added a commit that referenced this pull request Nov 12, 2026
Bring in main's integration configuration validation (#1455) and mediator
fallback (#1493) through Phase 2.
aram356 added a commit that referenced this pull request Nov 12, 2026
…igration

Conflicts keep both sides; #1493's mediator HTTP-error fallback moves onto run_mediator and its tests onto the scripted client, and #1455's checks sit beside the cut's config rejections.
aram356 added a commit that referenced this pull request Nov 12, 2026
The instream video pair is #1325 and #1437, not #1493, and the section now records which main PRs the Task 12 merges ported and that #1516 is still open.
@aram356 aram356 mentioned this pull request Nov 12, 2026
10 of 11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A non-2xx mediator response drops all winners instead of falling back to local ranking

4 participants