Skip to content

GH-27: Streaming (SSE) support for completion endpoints - #33

Merged
blainemotsinger merged 11 commits into
mainfrom
GH-27
Sep 20, 2026
Merged

blainemotsinger merged 11 commits into
mainfrom
GH-27

Conversation

@blainemotsinger

@blainemotsinger blainemotsinger commented Sep 20, 2026 •

Copy link
Copy Markdown
Member

Implements streaming (SSE) support for /v1/chat/completions and /v1/completions with stream: true.

Behavior

  • Responses are streamed chunk-by-chunk through a commit gate that holds the response uncommitted until the first chunk is flushed. Retry is allowed until that point; once committed, an upstream failure terminates the stream (clean EOF, no fabricated [DONE], never a 502 after commit).
  • Stalled streams that never deliver a first chunk are abandoned and retried on the next node after a fixed 10s first-byte deadline (streamFirstByteTimeout).
  • Non-streaming requests keep the existing buffered attempt behavior (byte-identical).
  • Client aborts before the first chunk are silent (no status/body written).

Changes

  • internal/api/stream.go (new): commitGate response writer, streamBody first-byte deadline + error capture, streamAttempt gate-wrapped reverse proxy
  • internal/api/proxy.go: extract shared newProxy constructor
  • internal/api/handler.go: stream: true dispatch branch with commit/retry/abort outcome loop; buffered path untouched
  • internal/api/stream_test.go + handler_test.go: unit and end-to-end coverage — gate commit semantics, deadline abort/disarm, reset-before-commit retry, stall deadline retry, truncated-after-commit (no 502), client abort, 201 empty stream, JSON error passthrough, happy-path flush
  • README.md: replace the "No streaming support" limitation with documented streaming semantics

Verification

  • make test, make vet, gofmt -l ., go build ./... all green
  • GOTOOLCHAIN=go1.22.12 go test ./... -race -count=1 (CI pin) — all packages pass
  • Streaming test stability: -race -count=10 green on both go1.26.3 and go1.22.12 (170/170 each)

closes #27

@blainemotsinger blainemotsinger self-assigned this Sep 20, 2026
@blainemotsinger
blainemotsinger merged commit f03a76d into main Sep 20, 2026
4 checks passed
@blainemotsinger
blainemotsinger deleted the GH-27 branch September 20, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming (SSE) support for completion endpoints

1 participant