[dhruv8sh]

Summary

• Turns pull sync into an identity provider that runs in the after-response phase from Collect server-side identity provider EIDs in a single page-load write #1487, so pull-synced partner IDs flow through the same coordinator and KV writes as other providers without adding page latency.
• Mirrors provider-owned bidstream UIDs to a JS-readable __Host-ts-eids-mirror cookie so publisher-side code can read them, on a best-effort basis.

Stacked on #1487; review only the commits on top of it.

Changes

Commit	Change
Run pull sync as an identity provider on navigations	New trusted-server-identity-providers/src/pull_sync.rs; removes core ec/pull_sync.rs / ec/pull_sync_marker.rs and the old post-send pull sync path in the Fastly adapter. Pull-sync providers are built with collect = "after_response", so they only run for an EC whose consented row already exists
Mirror provider-owned bidstream UIDs to a JS-readable cookie	New ec/mirror.rs; ec/finalize.rs, ec/cookies.rs, constants.rs write the __Host-ts-eids-mirror cookie
Make mirroring best effort and fix pull sync validation	Mirror failures no longer fail the response; the mirror cookie is expired only when the browser sent one; pull sync config validation fixes
Stop mirroring ownerless UIDs and re-pulling held pull-sync sources	Only owned UIDs are mirrored; ownerless batch records count as held for pull sync
Keep batch sync for pull-sync partners	Batch sync rejects only sources owned by a configured identity provider, so a partner with both pull sync and an api_token keeps batch sync
Set the mirror cookie only when its value changes	An unchanged mirror emits no Set-Cookie, so returning visitors' pages stay cacheable
Skip identity lookups when the rate limiter cannot count	A rate-limiter error now skips the lookup (as pull sync did before) for every provider
Reject the pull-sync kind in identity provider config	kind = "pull-sync" in [[ec.identity_providers]] is rejected; pull sync comes only from partner config with its token and rate limit
Refuse to key the template cache on the EID mirror cookie	__Host-ts-eids-mirror joins the identity cookies the template cache key refuses
Rotate pull-sync partners hourly	Partners beyond the per-navigation cap still get turns
Docs	Rate-limit rounding, the per-navigation cap and the mirror exposure callout in docs/guide/edge-cookies.md and configuration.md
Tests	Pull-sync results mirrored with their owner, non-navigations leaving the mirror untouched, withdrawn rows expiring it

Behaviour changes from main

• Pull sync timing. Pull sync now runs in the after-response phase, within identity_after_response_budget_ms, and only for an EC whose consented row already exists.
• pull_sync_concurrency is a cap per navigation. It used to be a batch width with every eligible partner pulled; partners now rotate hourly so each gets turns under the cap.
• Rate limits. pull_sync_rate_limit is enforced per minute with a minimum of one on Fastly, so the default of 10 per hour behaves like 60.
• Mirror exposure. Pull-sync UIDs now carry an owner, so for bidstream_enabled partners they reach the script-readable mirror cookie. Previously they lived only in KV.

Closes

Part of #1246 and #1245; the provider implementations that close them come in follow-up PRs.

Spec

Spec: reviewed separately; not included in this PR.

Test plan

• cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
• All eight clippy aliases
• cargo fmt --all -- --check
• Parity tests, native template build-digest test and clippy, scripts/lint-wasm-clock.sh
• JS tests, format and lint
• Docs format and Markdown prettier check
• ./scripts/test-cli.sh, domain lint against the base branch, CI Node pin guard, docs-proposal scripts
• HTML EOF checks (native harness, Axum and Fastly auction controls)
• Fastly EC integration tests (left to CI)
• Manual testing via fastly compute serve

Checklist

• Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
• Changes follow AGENTS.md conventions
• No unwrap() in production code — use expect("should ...")
• Uses log macros (not println!)
• New code has tests
• No secrets or credentials committed

[ChristianPavilonis]

Summary

This PR moves pull sync into after-response identity collection and adds a bounded, JavaScript-readable mirror of provider-written bidstream UIDs. One startup-validation issue needs addressing; there is also an opportunity to avoid a redundant KV read on steady-state navigations.

The inline comment is prose-only: the proposed fix and regression coverage touch lines outside the diff, so there are no one-click suggestions.

Blocking

🔧 wrench

• Reject malformed pull tokens during startup — see inline at crates/trusted-server-identity-providers/src/pull_sync.rs:59-60.

Cross-cutting / body-level findings

♻️ Avoid the follow-up KV read when all lookups were skipped

Introduced by crates/trusted-server-identity-providers/src/lib.rs:169-182; affected helper: crates/trusted-server-core/src/ec/after_response.rs:76-79.

When every pull-sync source already holds a UID, dispatch produces Skipped(NotDue) results. Those results are nonempty, so the follow-up helper rereads KV despite having nothing to persist. Moving existing pull-sync partners into this path adds an avoidable read on every eligible returning navigation.

Import LookupStatus and SkipReason alongside LookupResults, then replace the helper's empty-results guard with:

    if lookups.iter().all(|lookup| {
        lookup.entries.is_empty()
            && lookup.cleared.is_empty()
            && lookup.status != LookupStatus::Skipped(SkipReason::OptedOut)
    }) {
        return AfterResponseOutcome::Unchanged;
    }

This preserves writes that remember provider opt-outs.

Apply manually — the helper, imports, and regression test are outside the diff.

Verification

• A focused scratch reproduction confirmed that a newline-containing pull token passes runtime config validation but prevents construction of the coordinator for both an invalid and an otherwise valid partner.
• A counting-store scratch reproduction confirmed that all-NotDue results still cause one KV read in the follow-up helper.
• Both reproductions passed. Scratch changes were removed and the worktree was restored to the reviewed head.
• The proposed fixes have not been compiled or run through the full local gate.

CI Status

• integration tests: PASS
• browser integration tests: PASS
• integration tests (Fastly EC lifecycle): PASS
• trace runtime and browser acceptance: PASS
• cargo fmt: PASS
• cargo test (axum native): PASS
• cargo test (cross-adapter parity): PASS
• format-docs: PASS
• worker-deadlines (spin): PASS
• worker-deadlines (cloudflare): PASS
• ts dev lint domains (changed lines): PASS
• cargo test: PASS
• format-typescript: PASS
• cargo test (ts CLI, native) (macos-latest): PASS
• HTML EOF runtime (Fastly): PASS
• cargo check/build/test (spin native + wasm32-wasip1): PASS
• CLAUDE.md symlink guard: PASS
• docs-proposal scripts: PASS
• CI Node pin guard: PASS
• cargo test (ts CLI, native) (ubuntu-latest): PASS
• cargo check (cloudflare native + wasm32-unknown-unknown): PASS
• prepare integration artifacts: PASS
• vitest: PASS

No required checks were reported by the required-check query.

[ChristianPavilonis]

🔧 Reject malformed pull tokens during startup

A pull token containing a newline passes runtime config validation but fails here. Because provider construction collects into one Result (src/lib.rs:169-182), that single invalid token prevents every after-response provider from running. The request-time caller merely logs the failure.

Reject invalid resolved tokens in crates/trusted-server-core/src/ec/registry.rs during startup, while retaining the disabled-partner bypass:

    if require_nonempty_token
        && token_value.is_some_and(|value| {
            value.trim().is_empty()
                || http::HeaderValue::try_from(format!("Bearer {value}")).is_err()
        })
    {
        return Err(Report::new(TrustedServerError::Configuration {
            message: "ts_pull_token must be nonempty and valid in an HTTP Authorization header"
                .to_owned(),
        }));
    }

Add regression coverage for invalid enabled tokens and disabled partners. Validate resolved secret values at runtime config load, not secret-store key references during deploy validation.

Apply manually — the fix and regression tests touch lines outside this PR's diff hunks.

A focused scratch reproduction confirmed the startup-acceptance/coordinator-failure sequence with two partners. The proposed fix itself has not been compiled.

[aram356]

Summary

This PR moves pull sync onto the after-response provider path and adds a JavaScript-readable mirror of provider-written bidstream UIDs. The move keeps the live-row consent re-check and the bot gate. Before the mirror ships, it needs an operator opt-in and a per-vendor consent check. The after-response path also adds KV reads that the legacy dispatcher avoided.

3 of the 10 inline comments carry a one-click GitHub suggestion. Use Commit suggestion, or Add suggestion to batch to apply several at once. The other comments describe the fix in prose, because those changes span files or touch lines outside the diff.

Blocking

🔧 wrench

• Make the EID mirror opt-in, separate from bidstream_enabled: see inline at crates/trusted-server-core/src/ec/finalize.rs:90
• Withhold mirrored UIDs whose provider lacks TCF vendor consent: see inline at crates/trusted-server-core/src/ec/mirror.rs:77
• Skip the after-response row read when nothing is due: see inline at crates/trusted-server-adapter-fastly/src/app.rs:453
• Cover the enabled = false bypass of the double-claim check (suggestion): see inline at crates/trusted-server-core/src/config.rs:1455

❓ question

• Should pull-sync sources keep accepting browser UIDs?: see inline at crates/trusted-server-core/src/ec/registry.rs:274

Non-blocking

♻️ refactor

• Delete pull_enabled_partners; nothing calls it now: see inline at crates/trusted-server-core/src/ec/registry.rs:358
• Keep one definition of "runs pull sync": see inline at crates/trusted-server-core/src/ec/registry.rs:395

🤔 thinking

• The pull-sync cap is logged as skipped_rate_limited: see inline at crates/trusted-server-core/src/ec/coordinator/mod.rs:578

⛏ nitpick

• Describe where pull sync runs now (suggestion): see inline at docs/guide/edge-cookies.md:268
• The None case cannot happen (suggestion): see inline at crates/trusted-server-identity-providers/src/lib.rs:71

Cross-cutting / body-level findings

• 📝 "Behaviour changes from main" is missing items. In addition to the four already listed:
  • Browser ts-eids and sharedId UIDs are no longer stored for pull-sync sources. See the question at registry.rs:274.
  • Pull sync runs only on Fetch Metadata navigations that get a page-load permit. It no longer runs after /auction responses, on navigations without Fetch Metadata, or on prefetches.
  • Each pull used to have a 15-second first-byte timeout. Pull sync now shares the 2-second identity_after_response_budget_ms with every after-response provider, so a partner slower than that is never filled.
  • pull_sync_concurrency = 0 used to mean one request at a time (.max(1), with every partner still pulled). It now means no limit.
  • A steady-state returning navigation makes up to two more KV reads after the response. See the inline comment at app.rs:453 and the open thread on pull_sync.rs:60.
• 📝 Earlier findings are still open. ChristianPavilonis's two findings apply unchanged at this head, and that thread has no reply yet:
  • 🔧 malformed pull tokens (pull_sync.rs:60)
  • ♻️ the follow-up read when every result is NotDue

Verification

• Scratch tests at this head did three things:
  • reproduced the browser-ingest change (trusted-server-identity-providers, native target)
  • reproduced the mirror's vendor-consent gap (ec/mirror.rs, wasm32-wasip1 under Viceroy)
  • confirmed that the double-claim bypass works
• I applied each suggestion to a scratch copy, first alone and then together with the others:
  • The config test passed rustfmt, the new test itself, all six adapter clippy aliases, test-fastly, test-axum, test-cloudflare, test-spin and the parity suite.
  • The doc comment passed rustfmt and clippy-fastly.
  • The guide line passed prettier.
  • The scratch changes were then discarded.
• The prose fixes (mirror opt-in, vendor filter, any_due, runs_pull_sync) have not been compiled.

CI Status

• browser integration tests: PASS
• cargo check (cloudflare native + wasm32-unknown-unknown): PASS
• cargo check/build/test (spin native + wasm32-wasip1): PASS
• cargo fmt: PASS
• cargo test: PASS
• cargo test (axum native): PASS
• cargo test (cross-adapter parity): PASS
• cargo test (ts CLI, native) (macos-latest): PASS
• cargo test (ts CLI, native) (ubuntu-latest): PASS
• CI Node pin guard: PASS
• CLAUDE.md symlink guard: PASS
• docs-proposal scripts: PASS
• format-docs: PASS
• format-typescript: PASS
• HTML EOF runtime (Fastly): PASS
• integration tests: PASS
• integration tests (Fastly EC lifecycle): PASS
• prepare integration artifacts: PASS
• trace runtime and browser acceptance: PASS
• ts dev lint domains (changed lines): PASS
• vitest: PASS
• worker-deadlines (cloudflare): PASS
• worker-deadlines (spin): PASS

No required checks are reported for this branch.

[aram356]

🔧 Make the EID mirror opt-in, separate from bidstream_enabled

apply_mirror runs on every finalized response, and no setting turns it off. Once a row holds a provider-written UID for a bidstream_enabled partner, navigations set __Host-ts-eids-mirror, which any script on the page can read. With this PR that includes every pull-sync partner that bids; before, those UIDs stayed in KV. The guide's only opt-out is bidstream_enabled = false, which also removes the UIDs from user.eids. An operator therefore cannot keep a partner's UIDs in the bidstream and off the page.

An always-on default has two more costs:

• Nothing reads the cookie yet. crates/trusted-server-js has no reader, and Add identity providers for ID5, LiveIntent, LiveRamp, ConnectID and UID2 #1488 adds none.
• The value can reach 3,584 bytes and is sent on every same-origin request. rewrite_origin_request (publisher.rs:3471) forwards the Cookie header unchanged, so the publisher origin receives it too. With ts-eids (capped at 3,072 bytes by the Prebid integration), Trusted Server cookies come to about 6.7 KB before any publisher or CMP cookie. That is close to the 8 KB per-header default of nginx (large_client_header_buffers) and Apache (LimitRequestFieldSize).

Proposed fix (apply manually: touches settings.rs, ec/registry.rs, ec/mirror.rs and the guide). Add an explicit switch that defaults to off, preferably per partner, carry it onto PartnerConfig, and check it in build_mirror:

// settings.rs, EcPartner
/// Whether this partner's provider-written UIDs are mirrored to the
/// JS-readable `__Host-ts-eids-mirror` cookie.
#[serde(default, deserialize_with = "from_value_or_str")]
pub mirror_enabled: bool,

// ec/mirror.rs, build_mirror
.filter(|partner| partner.bidstream_enabled && partner.mirror_enabled)

Then change the guide's opt-out bullet to name it.

[aram356]

🔧 Withhold mirrored UIDs whose provider lacks TCF vendor consent

The coordinator skips a provider when the request's TCF string denies its tcf_vendor_id, but the mirror checks only EC and EID-persistence consent. Suppose a provider stored a UID while vendor consent was granted, and the user then denies that vendor. The UID keeps being written to a cookie that every page script can read. This is the same gap as the live_eids thread on #1487, through a new path to the page. The vendors in #1488 carry TCF vendor IDs, so it applies as soon as they land.

A scratch test in this module reproduces it. With Purposes 1 and 4 granted and vendor 9999 denied, tcf_vendor_consent_denied(&consent, 9999) is true, and build_mirror still returns MirrorCookie::Set with the stored UID.

Proposed fix (apply manually: build_mirror has no provider configs today, so pass the denied owners in from ec_finalize_response, which has settings). Skip a UID when its owner's vendor consent is denied:

// ec/finalize.rs, built before calling build_mirror
let denied: HashSet<&str> = settings
    .ec
    .identity_providers
    .iter()
    .filter(|config| {
        config
            .tcf_vendor_id
            .is_some_and(|vendor| tcf_vendor_consent_denied(consent, vendor))
    })
    .map(|config| config.id.as_str())
    .collect();

// ec/mirror.rs, the UID filter
.filter(|uid| {
    uid.owner()
        .is_some_and(|owner| !denied.contains(owner.as_str()))
        && uid.is_live(now)
})

Please add a regression test for the denied-vendor case.

[aram356]

🔧 Skip the after-response row read when nothing is due

This line sends every pull-sync deployment through run_after_response_lookups (main.rs:730). That function calls graph.load_snapshot (main.rs:761) before it dispatches. Finalize issues the follow-up permit when the page-load write "succeeded or had nothing to change" (finalize.rs:221-224), so a steady-state returning visitor reaches that read on every navigation.

The legacy dispatcher avoided this read on purpose. It filtered partners against the request snapshot first ("a dispatch with nothing to pull must not add a KV operation", base ec/pull_sync.rs:158-160), so a row that held every pull UID cost no KV operation. Now a returning visitor whose pull sources are all held costs two reads and no partner request: this one, plus the commit_after_response_lookups re-read from the open thread on pull_sync.rs:60.

Proposed fix (apply manually: touches coordinator/mod.rs and main.rs, outside this hunk; not compiled). Check the request's snapshot, which finalize leaves holding the committed row, and return before the live read when no after-response provider has a due source. A source held in that snapshot is still held in the live row unless consent was withdrawn, and a withdrawal blocks the lookup anyway.

// coordinator/mod.rs
/// Whether any provider has a due source in `entry`, ignoring consent and
/// rate limits, so a caller can skip a row read that cannot lead to a lookup.
#[must_use]
pub fn any_due(&self, entry: Option<&KvEntry>, now: UnixSeconds) -> bool {
    self.providers.iter().any(|registered| {
        !entry.is_some_and(|entry| entry.opted_out(&registered.id, now))
            && !Self::due_sources(registered, entry, now).0.is_empty()
    })
}

// main.rs, run_after_response_lookups, before `graph.load_snapshot`
let request_entry = ec_state.ec_context.kv_snapshot().entry_for(&lookups.ec_id);
if !coordinator.any_due(request_entry, UnixSeconds::new(current_timestamp())) {
    log::debug!("After-response lookups skipped: nothing due");
    return;
}

A NotRead or Failed request snapshot yields no entry, so every source counts as due and the live read still runs.

[aram356]

🔧 Cover the enabled = false bypass of the double-claim check

The new check in validate_identity_providers filters on config.enabled, but only the rejection is tested. The test below passes at this head, and it fails if a later change drops the filter.

Suggested change

	}
	}
	#[test]
	fn runtime_validation_accepts_disabled_provider_sharing_a_pull_sync_source() {
	let mut settings = settings_with_identity_provider(serde_json::json!({ "enabled": false }));
	settings.ec.partners = vec![
	serde_json::from_value(serde_json::json!({
	"name": "ids",
	"source_domain": "ids.example.com",
	"pull_sync_enabled": true,
	"pull_sync_url": "https://sync.example.com/pull",
	"pull_sync_allowed_domains": ["sync.example.com"],
	"ts_pull_token": "example-pull-token-32-bytes-minimum",
	}))
	.expect("should build partner"),
	];
	validate_settings_for_runtime(&settings)
	.expect("should accept a disabled provider sharing a pull-sync source");
	}

[aram356]

❓ Should pull-sync sources keep accepting browser UIDs?

This marks every pull-sync partner's source as provider-owned. apply_browser_ingest (write_plan.rs:94) skips provider-owned sources, so ts-eids and sharedId UIDs for these sources are no longer stored. On the base branch and on main, a missing pull-sync partner UID was filled from the browser. I ran a scratch test that builds the registry with partner_registry and plans a write with a browser UID for the partner's source. The UID is stored with pull_sync_enabled = false and dropped with pull_sync_enabled = true.

For configured providers, ownership keeps a UID cleared by an opt-out from being refilled from the browser. Pull sync never returns LookupOutcome::OptedOut, so that reason doesn't apply here. The PR already splits batch_sync_rejected from provider_owned so that batch sync keeps working for these partners.

Is dropping browser ingest intended, for example so that pull sync still runs and its owned UIDs reach the mirror? If it is, please list it under "Behaviour changes from main". If it isn't, a matching split for browser ingest would restore the old fill.

[aram356]

♻️ Delete pull_enabled_partners; nothing calls it now

Its only caller was the legacy dispatcher in ec/pull_sync.rs, which this PR deletes. The only remaining use is its own test, pull_enabled_partners_filters_correctly (line 806). Please remove both.

Apply manually: the function and its test are in separate hunks.

[aram356]

♻️ Keep one definition of "runs pull sync"

pull_sync_enabled && pull_sync_url.is_some() now appears in three places: here, in config.rs:387, and as the filter pair in pull_sync_providers (trusted-server-identity-providers/src/lib.rs:73-77). These copies decide which sources become provider-owned, which double claims are rejected, and which providers are built. If one copy changes without the others, a source can become provider-owned with no provider to fill it.

Separately, prepare_after_response_lookups (app.rs:453) builds a full IdentityProviderConfig, cloning the URL, domains and token, on every navigation just to ask whether one exists.

Proposed fix (apply manually: spans settings.rs, registry.rs, config.rs, the providers crate and the Fastly adapter):

impl EcPartner {
    /// Whether this partner runs pull sync as a provider, owning its source.
    #[must_use]
    pub fn runs_pull_sync(&self) -> bool {
        self.pull_sync_enabled && self.pull_sync_url.is_some()
    }
}

Use it in all three places, and in app.rs:

        || state.settings.ec.partners.iter().any(EcPartner::runs_pull_sync);

[aram356]

🤔 The pull-sync cap is logged as skipped_rate_limited

A partner past pull_sync_concurrency and a partner over its hourly limit both log outcome=skipped_rate_limited. The logs therefore can't show that the cap is too low. This is a new way for a partner to go unpulled, since pull sync used to reach every eligible partner. A separate SkipReason, for example ConcurrencyCap logged as skipped_concurrency_cap, would make it visible.

[aram356]

⛏ Describe where pull sync runs now

"As it did before it became a provider" no longer holds. The post-send dispatcher ran after any real-browser response whose request had read the row. That included /auction (auction/endpoints.rs:345 loads it) and navigations matched by the Accept: text/html fallback, prefetches among them. Each pull had a 15-second first-byte timeout (DEFAULT_FIRST_BYTE_TIMEOUT). Now pull sync runs only on Fetch Metadata navigations that get the page-load permit, inside the shared after-response budget. Suggested wording:

Suggested change

Pull sync runs as an identity provider on top-level page navigations, after the response has been sent, as it did before it became a provider. It uses the after-response budget (`identity_after_response_budget_ms`), so a slow partner never delays the page, and its result reaches the row in the follow-up write and is served from the next page load. Like other after-response lookups, it never creates a row. A pull-sync partner's source is provider-owned, so browser EID ingestion skips it. Batch sync still accepts the partner's mappings, as before. A source that already holds any live UID, including a batch-synced one, is not pulled again; a source with no UID is pulled at most `pull_sync_rate_limit` times per hour per EC. On Fastly that limit is enforced as a per-minute budget of at least one, so values below 60 behave like 60 per hour. At most `pull_sync_concurrency` partners are pulled per navigation (0 means no limit), unlike before, when every eligible partner was pulled in batches of that size. The partner order rotates every hour, so partners past the cap still get turns.

Pull sync runs as an identity provider after the response has been sent, on top-level page navigations that send Fetch Metadata (`Sec-Fetch-Dest: document` and `Sec-Fetch-Mode: navigate`) and are not prefetches. Before it became a provider, it also ran after `/auction` responses and on navigations without Fetch Metadata. It uses the after-response budget (`identity_after_response_budget_ms`, shared with the other after-response providers), so a slow partner never delays the page; a partner that answers after the budget is not filled, where each pull used to wait up to 15 seconds for a first byte. Its result reaches the row in the follow-up write and is served from the next page load. Like other after-response lookups, it never creates a row. A pull-sync partner's source is provider-owned, so browser EID ingestion skips it. Batch sync still accepts the partner's mappings, as before. A source that already holds any live UID, including a batch-synced one, is not pulled again; a source with no UID is pulled at most `pull_sync_rate_limit` times per hour per EC. On Fastly that limit is enforced as a per-minute budget of at least one, so values below 60 behave like 60 per hour. At most `pull_sync_concurrency` partners are pulled per navigation (0 means no limit), unlike before, when every eligible partner was pulled in batches of that size. The partner order rotates every hour, so partners past the cap still get turns.

[aram356]

⛏ The None case cannot happen

PartnerRegistry::from_config rejects pull_sync_rate_limit = 0 through validate_rate_limits, so NonZeroU32::new here always returns Some.

Suggested change

	/// the partner's `pull_sync_rate_limit` (`None` when it is 0).
	/// the partner's `pull_sync_rate_limit`, which partner validation keeps nonzero.