SSE Streaming for LLMs: Protocol Analysis, Hertz Implementation & Internal RPC Trade-offs

1. Why SSE for LLM Streaming

LLM inference produces tokens sequentially. To deliver a typewriter-like user experience, the server must stream tokens as they are generated. OpenAI chose Server-Sent Events (SSE) because it meets the functional need without the overhead of WebSocket and offers broad browser compatibility — SSE was defined in the WHATWG Web Applications 1.0 proposal (2004) and implemented in Opera by 2006.

2. Observing SSE in the Wild: DeepSeek Packet Capture

The author captures a real DeepSeek API call using curl with SSL key logging and inspects the TCP stream in Wireshark.

SSLKEYLOGFILE=./ssl_keys.log curl -v https://api.deepseek.com/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-xxoo" \
  -d '{
    "model": "deepseek-chat",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ],
    "stream": true
  }'

The response uses HTTP/1.1 with Content-Type: text/event-stream; charset=utf-8 and Transfer-Encoding: chunked. Each chunk contains one or more SSE events. Notably, DeepSeek wraps SSE inside HTTP chunked transfer, although the SSE specification cautions that chunking can harm reliability if the chunking layer is unaware of timing requirements.

Authors are also cautioned that HTTP chunking can have unexpected negative effects on the reliability of this protocol, in particular if the chunking is done by a different layer unaware of the timing requirements. If this is a problem, chunking can be disabled for serving event streams.

Events are separated by a blank line. Each event consists of name: value fields; DeepSeek uses only the data field, embedding JSON payloads. The spec also defines Last-Event-ID for resumability, but DeepSeek omits it — a pragmatic trade-off: the spec is comprehensive, implementations must choose.

3. Production-Grade SSE with cloudwego/hertz

The author critiques Spring WebFlux's reactive complexity and notes Go's goroutines and Node's async/await are better suited. Hertz (cloudwego/hertz) is used for both client and server.

3.1 Client Example

req.SetOptions(
  config.WithReadTimeout(5*time.Second),
  config.WithRequestTimeout(60*time.Second))
fmt.Println("Connecting to SSE server...")
if err := c.Do(context.Background(), req, resp); err != nil {
  fmt.Printf("Error sending request: %v
", err)
  return
}
if resp.StatusCode() != 200 {
  fmt.Printf("Unexpected status code: %d
", resp.StatusCode())
  return
}
r, err := sse.NewReader(resp)
if err != nil {
  fmt.Printf("Error creating SSE reader: %v
", err)
  return
}
defer r.Close()
fmt.Println("Connected to SSE stream. Receiving events...")
err = r.ForEach(ctx, func(e *sse.Event) error {
  fmt.Printf("Event received:
")
  fmt.Printf("  ID: %s
", e.ID)
  fmt.Printf("  Type: %s
", e.Type)
  fmt.Printf("  Data: %s

", string(e.Data))
  return nil
})

Key points: ReadTimeout controls the maximum idle time between events; RequestTimeout bounds the entire response duration.

The context ctx enables custom cancellation logic.

3.2 Server Example

lastEventID := sse.GetLastEventID(&c.Request)
hlog.CtxInfof(ctx, "Server Got LastEventID: %s
", lastEventID)
w := sse.NewWriter(c)
connClosed := ctx.Done()
for i := 0; i < 10; i++ {
  select {
  case <-connClosed:
    hlog.CtxInfof(ctx, "Client disconnected, stopping event transmission")
    return
  default:
    // continue
  }
  id := fmt.Sprintf("id-%d", i)
  data := fmt.Sprintf("Event data: %d at %s", i, time.Now().Format(time.RFC3339))
  hlog.CtxInfof(ctx, "Preparing to send event: %s, data: %s", id, data)
  err := w.WriteEvent(id, "message", []byte(data))
  if err != nil {
    hlog.CtxErrorf(ctx, "Error writing event: %v", err)
    break
  }
  time.Sleep(time.Duration(rand.Intn(10)) * time.Second)
}
w.Close()
hlog.CtxInfof(ctx, "SSE event transmission completed")

Go's context elegantly handles I/O cancellation. The server checks ctx.Done() each iteration to detect client disconnects.

3.3 Keep-Alive for Proxies

Proxies may close idle connections. Hertz provides WriteKeepAlive which sends a comment line : keep-alive to keep the channel open:

// WriteKeepAlive writes a comment line with "keep-alive" to the response body.
// It keeps the underlying connection alive, which is useful when using proxy servers.
func (w *Writer) WriteKeepAlive() error {
  return w.WriteComment("keep-alive")
}

4. Internal Service-to-Service Streaming RPC

Beyond browsers, internal services also need streaming — e.g., an A/B routing layer that forwards requests to different LLM APIs, processes the stream, and returns results to the caller.

The question: should internal calls use gRPC streaming or stick with SSE end-to-end?

The author prefers SSE throughout because:

gRPC's purported advantages (lower protocol overhead, better performance) do not materialize in practice.

gRPC adds significant complexity compared to HTTP/SSE.

CloudWeGo's Kitex recently introduced StreamX, a simpler internal streaming protocol. The author plans a separate comparison.

References

SSE Specification:

https://html.spec.whatwg.org/multipage/server-sent-events.html

cloudwego/hertz: https://github.com/cloudwego/hertz Example code: https://gitee.com/shichaoyuan/some-stream Spring WebFlux:

https://docs.spring.io/spring-framework/reference/web/webflux.html

gRPC Server Streaming:

https://grpc.io/docs/what-is-grpc/core-concepts/#server-streaming-rpc

Kitex StreamX:

https://www.cloudwego.io/zh/docs/kitex/tutorials/basic-feature/streamx/