Why Your Elasticsearch Client Doubles Bandwidth and How to Stop It

Problem Overview

In a high‑throughput log‑writing scenario the service reports a write rate of 50 MB/s (≈1 TB per day), yet network monitoring shows 100 MB/s outbound traffic, fully saturating the NIC and causing packet loss.

Investigation

TCP retransmission was normal.

SSL handshake overhead was not the cause.

Switch statistics confirmed the bandwidth spike.

Capturing traffic with

tcpdump -i eth0 port 9200 -A -s 0 -c 100 -w bandwidth_leak.pcap

revealed that each POST request’s body was transmitted twice.

Root Cause – Non‑Preemptive (Passive) Authentication

The client follows RFC 2617’s “passive authentication” flow: it first sends the request without an Authorization header, receives a 401 Unauthorized response, then retries with credentials. This results in an ineffective first transmission that still consumes full bandwidth.

# Example of captured packets
POST /_bulk (no Auth header) → 401 Unauthorized
POST /_bulk (with Auth header) → 200 OK

Solution – Enable Pre‑emptive Authentication

Java Client

Configure a CredentialsProvider and inject it into the low‑level RestClientBuilder so that the Authorization header is sent on the first request.

// Prepare credentials provider
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
    new UsernamePasswordCredentials("user", "password"));

// Configure RestClient with pre‑emptive auth
RestClient restClient = RestClient.builder(new HttpHost("es-host", 9200))
    .setHttpClientConfigCallback(httpClientBuilder ->
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))
    .build();

ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
ElasticsearchClient client = new ElasticsearchClient(transport);

Older RestHighLevelClient

Either enable the internal auth cache or set the Authorization header directly.

// Option 1: Use CredentialsProvider (same as above)
// Option 2: Hard‑code header
Header[] defaultHeaders = new Header[]{
    new BasicHeader("Authorization", "Basic " +
        Base64.getEncoder().encodeToString("u:p".getBytes()))
};
RestClientBuilder builder = RestClient.builder(new HttpHost("es-host", 9200))
    .setDefaultHeaders(defaultHeaders);

Python Client

In the modern elasticsearch v8 client, use basic_auth, which enables pre‑emptive auth by default.

from elasticsearch import Elasticsearch

client = Elasticsearch(
    "http://es-host:9200",
    basic_auth=("user", "password")
)

For older versions or when unsure, manually add the Authorization header.

import base64, requests

token = base64.b64encode(b"user:password").decode("ascii")
headers = {"Authorization": f"Basic {token}"}
response = requests.post(url, data=big_payload, headers=headers)

Pro Tips

Prefer API keys over basic auth for better performance and security (when supported).

In serverless environments that lack API‑key support, configure pre‑emptive basic auth as shown.

Monitoring the Issue

Use end‑to‑end request metrics to compare 401 and 200 curves; a 1:1 overlap indicates the double‑send problem.

Inspect access logs for user_agent or remote_ip fields to pinpoint the offending service.

Fixing the authentication configuration can halve the effective bandwidth consumption and reduce traffic costs dramatically.