Skip to content

Recreate expired UDP sessions without losing the first datagram - #108

Merged
SajjadPourali merged 3 commits into
narrowlink:mainfrom
szonov:fix/udp-session-replacement
Nov 2, 2026
Merged

SajjadPourali merged 3 commits into
narrowlink:mainfrom
szonov:fix/udp-session-replacement

Conversation

@szonov

@szonov szonov commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

When a UDP stream is dropped, its sender can remain in the session table until the asynchronous destroy notification is processed. A datagram arriving in that interval is sent to a closed receiver and lost. This is the scenario reported by @ple1n in #81, which also links the implementation used by nsproxy.

Recover the undelivered packet from SendError, remove the dead entry, and create a replacement stream with that packet as its first payload. Keep the existing asynchronous cleanup, but remove an entry on a destroy notification only if its current sender is closed. This prevents an old stream's delayed notification from deleting a live replacement.

Close the UDP receiver explicitly before sending its destroy notification. This ensures cleanup observes a closed sender even when another runtime worker processes the notification before Drop returns.

Extract the existing registration path so initial and replacement streams use the same setup. Extract the removal handler used by the event loop so the deterministic race test exercises production cleanup code rather than repeating its condition.

Fixes #81.

Tests

  • receiver_is_closed_before_destroy_notification_wakes_waiter: registers a custom waker on the destroy receiver and checks channel closure synchronously during notification delivery, before the UDP destructor returns. Fails without the explicit close() and passes with it.

  • dead_udp_session_preserves_payload_and_survives_stale_removal: closes a stream, queues its destroy notification, dispatches the next datagram before applying that notification, reads and checks its complete payload, runs the production removal handler, and checks the next datagram still reaches the same replacement. Finally checks cleanup of the replacement itself.

  • stale_udp_removal_does_not_remove_a_live_session: independently catches unconditional removal of a live sender.

  • dropped_udp_session_is_cleaned_without_more_traffic: runs the real IP-stack event loop with an in-memory device and verifies that dropping a stream releases the table's sender without more device input.

  • expired_udp_session_is_cleaned_without_more_traffic: checks the same cleanup after the configured inactivity timeout is observed and the caller drops the expired stream. Uses Tokio's paused clock.

On unpatched upstream, dead_udp_session_preserves_payload_and_survives_stale_removal and stale_udp_removal_does_not_remove_a_live_session fail: dispatch returns SendError, and the stale notification removes the live entry. The two cleanup tests also protect existing cleanup behavior from regressions; they pass before and after the fix.

Initial validation on upstream 44bb34fe50bd9c04290b4b4f9bbeece1626a95d4 on macOS arm64 with Rust 1.99.0:

  • cargo fmt -- --check
  • cargo test --offline --locked: 40 unit tests and 23 documentation tests pass.
  • cargo clippy --offline --locked --all-targets -- -D warnings

The same commit and Cargo.lock were also validated on Linux ARM64 (Debian 12, Rust 1.98.1): all 40 unit tests and 23 documentation tests pass, as do formatting, Clippy with all targets/features and -D warnings, and the examples/tests build. Checks ran offline in a container with networking disabled.

No changes to dependencies or public APIs. No live TUN/network test was performed.

Native Linux AMD64 validation also passes on Debian 13 with Rust 1.98.1, using the same commit and Cargo.lock: 40 unit tests, 23 documentation tests, formatting, Clippy for all targets/features with -D warnings, and the examples/tests build.

Validation after review update

Merged current upstream 8593d76 (including #109 and #110) into this branch. On macOS arm64 with Rust 1.99.0, head c3dc279 passes:

  • cargo fmt -- --check
  • cargo test --offline --locked: 52 unit tests and 23 documentation tests
  • cargo clippy --offline --locked --all-targets --all-features -- -D warnings
  • cargo build --offline --locked --examples --tests
  • git diff --check

The new notification-order test was also run before the fix and failed as expected. Linux validation above applies to the original revision; this review update has only been validated on macOS.

@SajjadPourali
SajjadPourali self-requested a review November 2, 2026 04:28
Comment thread src/lib.rs
// A dead stream may already have been replaced while its destroy notification
// was queued. Never remove a live replacement on behalf of the old stream.
if sessions.get(&network_tuple).is_some_and(PacketSender::is_closed) {
sessions.remove(&network_tuple);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it would be better to close the stream in the drop prior to sending the destroy notification.

impl Drop for IpStackUdpStream {
    fn drop(&mut self) {
        self.stream_receiver.close();

        if let Some(messenger) = self.destroy_messenger.take() {
            let _ = messenger.send(());
        }
    }
}

IpStackUdpStream::drop() sends its destroy notification before Rust drops its receiver field. On a multithreaded runtime, the watcher and stack loop can process that notification while the receiver is still open. The new is_closed() check then skips removal. Once destruction finishes, no further notification arrives

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, you're right: the notification could be handled before Rust drops the receiver field, causing the closed-sender guard to skip cleanup. Fixed in c3dc279 by closing the UDP receiver before sending the notification, as suggested.

Added a deterministic regression test that registers a custom waker on the destroy receiver and checks sender.is_closed() synchronously when the notification wakes it, before the destructor returns. It fails without close() and passes with it. The stale-notification guard remains in place to protect a live replacement.

Also merged current upstream main. On macOS arm64, all 52 unit tests and 23 documentation tests pass, along with formatting, Clippy for all targets/features with -D warnings, and the examples/tests build.

@SajjadPourali
SajjadPourali merged commit abed080 into narrowlink:main Nov 2, 2026
2 checks passed
@szonov
szonov deleted the fix/udp-session-replacement branch November 2, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

There is a critical error in UDP handling. It drops packets. I have fixed it in my fork.

2 participants