Skip to content

Fix use of freed memory when a named pipe read times out (JNA) - #102

Open
lrytz wants to merge 1 commit into
sbt:2.xfrom
lrytz:pr/jna-win-overlapped
Open

lrytz wants to merge 1 commit into
sbt:2.xfrom
lrytz:pr/jna-win-overlapped

Conversation

@lrytz

@lrytz lrytz commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Windows reads and writes named pipes asynchronously, and keeps using the buffer until the operation is complete. With JNA, when a read with a timeout (SO_TIMEOUT) timed out, the code gave up on the read without cancelling it. The buffer is managed by the garbage collector, so it could be freed while Windows still used it: data arriving later was written into freed memory, and lost.

Now a read that times out is cancelled, and the code waits until Windows is done with it. If data arrived just before the cancel, the read returns it.

Writes had a similar problem: when a write couldn't complete right away, Windows could read the data from a copy that JNA had already freed. Now the data stays alive until the write is complete. Also, reading or writing zero bytes no longer throws.

@lrytz
lrytz force-pushed the pr/jna-win-overlapped branch from 6a91294 to a71cd87 Compare November 4, 2026 14:51
@lrytz lrytz mentioned this pull request Nov 4, 2026
@lrytz
lrytz force-pushed the pr/jna-win-overlapped branch 3 times, most recently from 4dc8fee to 44ec375 Compare November 4, 2026 19:55
@lrytz

lrytz commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@retronym review..?

@retronym retronym left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix looks correct. The old timeout path was worse than "data lost": the kernel still owned both readBuffer and the OVERLAPPED, so once the Cleaner freed them the completion wrote data into the freed buffer and status/byte-count into the freed struct, i.e. heap corruption. The abandoned read also stayed queued ahead of the next ReadFile, which is where the lost-message symptom came from.

Nits, none blocking:

  • readBuffer.close() only runs on the success path. Every other exit is after GetOverlappedResult has returned, or after a synchronous ReadFile failure with nothing outstanding, so wrapping it in try/finally like write() is safe and makes the invariant obvious.
  • Passing olap.getPointer() to CancelIoEx rather than NULL matters: sbt has a writer thread on the same handle, and NULL would cancel its write too. Worth a comment.
  • If CancelIoEx fails for a reason other than ERROR_NOT_FOUND, the following GetOverlappedResult(wait=TRUE) has no timeout. Only plausible during a concurrent close, where it fails fast anyway, so fine.
  • Worth stating in the description: a read that timed out can now return data if it arrived during the cancel window.
  • The ByteBuffer overload of WriteFile in Win32NamedPipeLibrary is now unused.

Follow-up (separate PR): the JNI readNative has related problems in a different shape. Its error macros don't return, so on timeout it sets SocketTimeoutException pending and falls through to GetOverlappedResult, which does wait for the cancel. But CancelIoEx(handle, NULL) cancels every operation on the handle, including the other thread's write; the ERROR_OPERATION_ABORTED failure then replaces the timeout with a plain IOException; and every failure path keeps making JNI calls with an exception pending. writeNative also leaves its OVERLAPPED uninitialised.

Branches with suggested changes, feel free to cherry-pick:

  • review/102-nits: the nits above (try/finally, CancelIoEx comment, unused overload).
  • review/102-jni-followup: the JNI fix plus JNI variants of the two new tests. Written without a Windows toolchain, so it needs a Windows CI run.

Windows reads and writes named pipes asynchronously, and keeps using the
buffer until the operation is complete. With JNA, when a read with a
timeout (SO_TIMEOUT) timed out, the code gave up on the read without
cancelling it. The buffer is managed by the garbage collector, so it
could be freed while Windows still used it: data arriving later was
written into freed memory, and lost.

Now a read that times out is cancelled, and the code waits until Windows
is done with it. If data arrived just before the cancel, the read
returns it.

Writes had a similar problem: when a write couldn't complete right away,
Windows could read the data from a copy that JNA had already freed. Now
the data stays alive until the write is complete. Also, reading or
writing zero bytes no longer throws.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lrytz
lrytz force-pushed the pr/jna-win-overlapped branch from 44ec375 to 105b50b Compare November 11, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants