Add syslog stream and datagram listeners - #9
Merged
Merged
Conversation
Adds two syslog transports to the server alongside HTTP, both delivering complete messages to a Ruby handler on the existing worker threads: - a stream listener (Unix socket or TCP) framed with RFC 6587 octet counting and a newline fallback, with optional PROXY v2 parsing to recover the peer address, and - a UDP listener with SO_REUSEPORT where one datagram is one message, with receive-buffer truncation detected and dropped. Undelivered messages are bounded per connection and overall; a refused message stalls stream reads and is retried, while a refused datagram is dropped and counted. Listener readiness and transport counters are exposed as syslog_listening? and syslog_stats. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The worker fast path always drained requests first, so sustained request load could starve syslog delivery indefinitely and stall every sender behind the per-connection backpressure. The two channels now alternate which is asked first, and the blocking select already picks uniformly between them. Workers with no syslog listener configured keep the original single-channel path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Licence: the framing module, including its test data, is a port of Vector 0.48.0's octet counting decoder, so the file, the README and the licence file record that it is MPL-2.0 rather than MIT. Framing: a read boundary landing on a digit inside an oversize body no longer restarts octet counting and hands that body over as messages. Delivery: the per-connection queue is bounded separately from the listener-wide one, so a stalled sender cannot take every credit; the handler is given a stable message id and attempt number, its exceptions are counted apart from refusals, and retries end when shutdown starts. Messages the drain could not admit are counted instead of vanishing. Lifecycle: a cancellation token replaces the per-connection broadcast so a connection accepted in the same round as the shutdown still sees it, and stop() takes owned handles before releasing the GVL. Connection and idle limits bound stream resources, and datagram receive errors back off. A PROXY header naming a Unix or unspecified family falls back to the transport peer, with read failures counted separately. Workers may take several syslog messages per request, so a busy request queue cannot pace syslog delivery. The handler now reports :stream or :datagram, a peerless stream gives nil, the datagram buffer keys are named for what they bound, and the handler stays reachable through the server object rather than a permanent GC root. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Requests and syslog messages now reach Ruby the same way: the block given to run_worker receives either a Request, which it answers with a Response, or a SyslogMessage, which it answers with an admission verdict. The separate configured handler, and the GC root it needed, are gone; delivery, limits, retries and accounting are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The framing module is MPL-2.0 while the rest is MIT, so the gem metadata lists both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
alistairjevans
changed the title
A block that answers a syslog message with a response was not giving a verdict, but its response read as truthy and admitted the message. That now logs once and refuses, counted as a handler error, and the failure log names the message and its transport. The message object is freed as soon as it is collected and reports its payload to the GC, its string is built without revalidating bytes the framer already checked, and inspect prints the transport as the reader returns it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The README described the removed callable, and the extension manifest named only one of the two licences the gem declares. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds syslog transport to the server alongside HTTP: a stream listener (Unix socket or TCP, optional PROXY protocol v2 for the peer address) framed with RFC 6587 octet counting and a newline fallback, and a datagram listener (UDP, SO_REUSEPORT) where one datagram is one message.
Messages are delivered on the existing worker threads with the same GVL discipline as requests: the block passed to
Server#run_workerreceives either aHyperRuby::Request(returns aResponse, as before) or aHyperRuby::SyslogMessage(message,peer_ip,transport,received_at_ns,message_id,attempt) and returns whether the message was admitted. A refused stream message is retried with a bounded delay while that connection's reads stall; a refused datagram is dropped and counted.Details
ext/hyper_ruby/src/syslog/framing.rs, MPL-2.0, noted inLICENSE.txtand the gemspec's licences), with one documented divergence: a read boundary landing on a digit inside a discarded oversize body no longer restarts counting.syslog_work_ratiomessages per request (default 4); servers without a syslog listener run the unchanged HTTP-only loop.Server#syslog_listening?andServer#syslog_statsexpose readiness and counters.lib/hyper_ruby.rb.Tests
cargo test: 28 Rust tests (framing at every byte boundary, PROXY parsing, oversize and UTF-8 handling).rake test: 67 runs; the syslog suite covers TCP and UDP loopback, split writes, oversize frames, invalid UTF-8, PROXY v2, backpressure isolation between connections, retry ids and attempts, delivery under HTTP load, and clean shutdown. The one failing test,TestBackpressure#test_backpressure_handling, fails identically onmaster.