Envoy Outbound Request Flow: Listener Startup, ReusePort Configuration, and Connection Establishment

Envoy is a high‑performance network proxy designed for service mesh environments, running alongside applications to provide platform‑agnostic networking features such as routing and traffic control. This series examines Envoy’s source code, focusing on outbound request handling, listener startup, and connection establishment.

Outbound Direction – Listener Startup

Listeners are obtained via xDS or static configuration. If a listener is bound to a port, Envoy calls libevent to bind and sets ListenerImpl::listenCallback as the callback.

void ListenerManagerImpl::addListenerToWorker(Worker& worker,
    absl::optional<uint64_t> overridden_listener,
    ListenerImpl& listener,
    ListenerCompletionCallback completion_callback) {
  if (overridden_listener.has_value()) {
    ENVOY_LOG(debug, "replacing existing listener {}", overridden_listener.value());
    worker.addListener(overridden_listener, listener,
        [this, completion_callback](bool) -> void {
          server_.dispatcher().post([this, completion_callback]() -> void {
            stats_.listener_create_success_.inc();
            if (completion_callback) { completion_callback(); }
          });
        });
    return;
  }
  // ... (additional listener setup code) ...
}

ReusePort Support

Enabling SO_REUSEPORT allows multiple server sockets to listen on the same port, distributing connections across threads based on a hash of the TCP 4‑tuple. Envoy can enable this feature in the listener configuration, but hot‑restart requires kernel version 4.19‑rc1 or newer.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: reuseport
  namespace: testhl
spec:
  workloadSelector:
    labels:
      app: asm-0
  configPatches:
    - applyTo: LISTENER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          portNumber: 15001
          name: "virtualOutbound"
      patch:
        operation: MERGE
        value:
          reuse_port: true

After applying the filter and restarting the pod, the listener on port 15001 shows the reuse‑port flag enabled, resulting in a more balanced connection distribution across threads.

Connection Establishment

The dispatcher receives the request via libevent and invokes ListenerImpl::listenCallback . The client connection triggers the OriginalDstFilter , which retrieves the original destination address using getsockopt with SO_ORIGINAL_DST .

Network::Address::InstanceConstSharedPtr OriginalDstFilter::getOriginalDst(Network::Socket& sock) {
  return Network::Utility::getOriginalDst(sock);
}

sockaddr_storage orig_addr;
memset(&orig_addr, 0, sizeof(orig_addr));
socklen_t addr_len = sizeof(sockaddr_storage);
int status;
if (*ipVersion == Address::IpVersion::v4) {
  status = sock.getSocketOption(SOL_IP, SO_ORIGINAL_DST, &orig_addr, &addr_len).rc_;
} else {
  status = sock.getSocketOption(SOL_IPV6, IP6T_SO_ORIGINAL_DST, &orig_addr, &addr_len).rc_;
}
if (status != 0) { return nullptr; }
return Address::addressFromSockAddr(orig_addr, 0, true);

Envoy then searches for the appropriate virtual listener using getBalancedHandlerByAddress , creates a ServerConnectionImpl via dispatcher.createServerConnection , and registers read/write events with edge‑triggered epoll.

auto server_conn_ptr = parent_.dispatcher().createServerConnection(
    std::move(socket), std::move(transport_socket), *stream_info);

Network::ServerConnectionPtr DispatcherImpl::createServerConnection(
    Network::ConnectionSocketPtr&& socket,
    Network::TransportSocketPtr&& transport_socket,
    StreamInfo::StreamInfo& stream_info) {
  ASSERT(isThreadSafe());
  return std::make_unique
(
      *this, std::move(socket), std::move(transport_socket), stream_info, true);
}

For HTTP listeners, the envoy.http_connection_manager filter is added to the filter chain, enabling request processing via the on_read callback.

void FilterManagerImpl::addReadFilter(ReadFilterSharedPtr filter) {
  ASSERT(connection_.state() == Connection::State::Open);
  ActiveReadFilterPtr new_filter(new ActiveReadFilter{*this, filter});
  filter->initializeReadFilterCallbacks(*new_filter);
  LinkedList::moveIntoListBack(std::move(new_filter), upstream_filters_);
}

When a new connection is added to the event loop, a write event may fire immediately, but without data in the write buffer no bytes are sent.

void CodecClient::onEvent(Network::ConnectionEvent event) {
  if (event == Network::ConnectionEvent::Connected) {
    ENVOY_CONN_LOG(debug, "connected", *connection_);
    connection_->streamInfo().setDownstreamSslConnection(connection_->ssl());
    connected_ = true;
  }
  // ... handling of remote close, stream resets, etc. ...
}

Overall, the article demonstrates how Envoy sets up listeners, leverages SO_REUSEPORT for better load balancing, retrieves original destination addresses, and processes connections through a series of well‑structured C++ components.