Conversation
Since Spring Cloud Gateway 4.1.2 the outbound chain is assembled in NettyRoutingFilter#filter but subscribed later, so the v412x plugin parked the request's ContextSnapshot on the HttpClient returned by NettyRoutingFilter#getHttpClient. That method returns the shared this.httpClient bean unless a connect timeout is configured, so the write lands on one object for the whole JVM while the read, HttpClientConnect#duplicate through HttpClient#headers, happens at subscription time. Any request reaching getHttpClient in between overwrites the snapshot, and the outbound sw8 header carries a context that belongs to another request. Hold the snapshot on a client derived per request instead. HttpClient#headers duplicates the client and copies the header map only, which Spring Cloud Gateway copies again one line later, so a chain that already duplicates on headers(), request() and uri() gains a single duplication and no observer. The shared bean is never written again, so HttpClientConnectDuplicateV412Interceptor cannot carry a stale value forward, and untraced requests derive nothing because the isActive() guard runs first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on the report in apache/skywalking#14095.
Problem
Since Spring Cloud Gateway 4.1.2,
NettyRoutingFilter#filteronly assembles the outbound chain; the chain is subscribed later:That is why
v412xcarries aContextSnapshotforward instead of reading the thread context atsend()the wayv4xdoes. The snapshot was parked on theHttpClientreturned byNettyRoutingFilter#getHttpClient— but that method returns the sharedthis.httpClientbean itself unless a connect timeout is configured, so the write lands on one object for the whole JVM lifetime, while the read (HttpClientConnect#duplicateviaHttpClient#headers) happens at subscription time.Any request that reaches
getHttpClientin between overwrites the snapshot, and the earlier request's outboundsw8header then carries a context that belongs to another request. The downstream service joins the wrong trace.Fix
Hold the snapshot on a client derived per request.
HttpClient#headersduplicates the client and copies the header map only, which Spring Cloud Gateway copies again one line later, so a chain that already duplicates onheaders(),request()anduri()gains a single duplication and no connection observer.doOnRequestwould also force the copy, but it flips reactor-netty off itsdefaultConnectionObserver()fast path and installs anHttpClientDoOnobserver on every request, so it is not used here.After this change the shared bean is never written, so
HttpClientConnectDuplicateV412Interceptorreadsnullfrom it and no stale value can be carried forward. Every node of the chain is now per request. Untraced requests derive nothing, because theContextManager.isActive()guard runs first.Tests
NettyRoutingGetHttpClientV412InterceptorTestnow asserts that the shared client is never stamped and that two requests get distinct snapshots. Both assertions fail against the previous interceptor.A note for the reporter
This fixes a real cross-request bleed, but it may not be the whole of what apache/skywalking#14095 observes: a concurrent overwrite yields a sibling request's trace from milliseconds ago, not one from ~832 seconds ago on the same thread. If the stale
sw8survives this patch, the remaining suspect is a leakedTracingContexton the event loop —TraceSegment#relatedGlobalTraceonly replaces the trace id while it is still aNewDistributedTraceId, so a reused leaked context keeps emitting its original trace id. Local verification against a real gateway is very welcome.