Production-Ready Elasticsearch Security Hardening: TLS, Authentication, and High‑Concurrency Architecture with INFINI Gateway

Why a Gateway Is Required

Many teams expose Elasticsearch directly on port 9200 and focus only on query speed and write stability. This ignores four fatal problems: clear‑text traffic, coarse authentication, uncontrolled traffic bursts (bulk writes, deep pagination, large aggregations, fuzzy queries), and lack of isolation between data and access layers. Elasticsearch is best used as a search, inverted‑index, and vector‑retrieval engine, not as an external security boundary.

Architecture Upgrade Perspective

Native Elasticsearch Security is valuable as a last line of defense , but a gateway provides a front‑line boundary. The comparison table shows that the gateway operates at the access layer, is transparent to ES, offers native QPS/path‑level rate limiting, records client IP and account, and runs in an independent process for fault isolation.

Recommended Production Security Layers

Layer 4 Business Identity: Basic Auth / Token / SSO / LDAP

Layer 3 Transport Security: TLS termination, mTLS, certificate rotation

Layer 2 Traffic Governance: rate limiting, concurrency control, circuit breaking, black/white lists

Layer 1 Request Security: path control, DSL risk filtering, audit

Layer 0 Data Protection: ES native RBAC, index permissions, network ACL

The recommended combination is:

Gateway (handles ~80% of entry‑point governance)
+ ES native Security (fine‑grained RBAC and fallback)
+ Network ACL / Security Groups (infrastructure isolation)

INFINI Gateway Core Principles

The gateway is more than a reverse proxy; it provides an orchestrable filter chain for Elasticsearch‑specific traffic.

Request Processing Model

Entry → Router → Flow → Filter Chain → Elasticsearch Backend

Entry

: listens on a port, handles TLS, concurrency, network params. Router: dispatches based on path, method, header, host. Flow: a container for an ordered set of filters. Filter: smallest governance unit (auth, rate‑limit, audit, forward). Elasticsearch Backend: connects to the ES cluster, supports node discovery and role‑based routing.

Why This Model Suits Production

Uniform rules – all clients (apps, scripts, Kibana, Beats) pass the same gateway.

Independent upgrades – rate limiting, audit, auth, certificate rotation can evolve without touching ES or applications.

Fault isolation – authentication failures, TLS handshakes, malicious queries are handled before they consume ES resources.

Production‑Ready Architectures

Minimal Viable Architecture (QPS < 5 000)

Client / Kibana / Script
   | HTTPS :8443
   | INFINI Gateway
   | HTTP :9200 (internal)
   | Elasticsearch

Key points: expose only the gateway externally, keep ES internal, enforce unified authentication, and route all requests through the filter chain.

Standard Production Architecture (QPS 5 000‑50 000)

+----------------------+   +-------------------+
|  SLB / NLB / Ingress | → |   GW‑01           |
+----------+-----------+   +--------+----------+
           |                     |
           +---------------------+   (replicated gateways)
                                 |
                                 v
                         +-----------------------+
                         | Elasticsearch Cluster |
                         +-----------------------+

Recommendations: at least two gateway replicas, load‑balancer in front, configuration managed via Helm values, ConfigMap, or GitOps, and secret‑managed TLS certificates.

Advanced Multi‑Cluster Architecture

Supports billions of daily requests, multi‑tenant isolation, read/write separation, and cross‑region traffic routing. The gateway can route writes to a primary cluster and reads to nearby clusters, with optional traffic mirroring for gray‑release testing.

Kubernetes Production Deployment

When moving to cloud‑native environments, ensure the following:

Multiple replicas for high availability.

Horizontal pod autoscaling based on CPU, active connections, 5xx rate, auth‑failure rate, and request queue time.

Stateless gateway pods with configuration supplied via Helm values or ConfigMap.

Certificate management via cert‑manager for automatic issuance and rotation.

Management API exposed only as a ClusterIP service, protected by network policies.

High‑Availability & Scalability Design

Because the gateway is stateless, horizontal scaling is straightforward: each instance loads identical config and shares the same ES backend addresses. Load balancers distribute traffic evenly.

Read/Write Separation

Separate entry points for bulk/write APIs ( /_bulk, /_doc, /_update) and read APIs ( /_search, /_msearch, /_count) to prevent write storms from degrading query latency.

Request Security Governance

High‑risk requests often come from legitimate credentials but consume excessive resources. The gateway blocks them early, saving ES resources.

Deep pagination ( "from":\s*[1-9]\d{4,}) → 403 with message “Deep pagination is forbidden, use search_after”.

Wildcard or regexp queries → 403 with clear messages.

Large terms aggregations or multi‑index scans are throttled via path‑level rate limits.

Bulk write size is limited per IP or per account.

High‑Concurrency Engineering

Key parameters to tune: max_concurrency: total entry‑point concurrency. max_connection_per_node: connections per ES node. timeout: backend request timeout.

Path/IP/account‑level rate‑limit rules.

Example capacity calculation: 3 ES nodes × 1 500 stable active requests × 0.7 safety margin ≈ 3 150 concurrent requests. Gateway limits should be set accordingly.

Real‑World Scenarios

Log Platform Bulk‑Write Protection

Problem: tenant spikes cause oversized _bulk packages, exhausting ES connections and slowing queries.

flow:
  - name: write_flow
    filter:
      - basic_auth:
          valid_users:
            ingest_core: ${INGEST_CORE_PASS}
            ingest_normal: ${INGEST_NORMAL_PASS}
      - request_path_limiter:
          rules:
            - pattern: "/_bulk"
              max_qps: 15000
              group: ip
      - elasticsearch:
          elasticsearch: prod
          timeout: 20s
          max_connection_per_node: 4000

Result: core writes are isolated, ES connections stay bounded, and query traffic is no longer throttled by bulk storms.

Search Platform DSL Risk Mitigation

Problem: unrestricted DSL allows deep pagination, wildcard, and massive index scans that overload the cluster.

flow:
  - name: read_flow
    filter:
      - basic_auth:
          valid_users:
            app_search: ${APP_SEARCH_PASS}
      - context_filter:
          action: redirect_flow
          flow: reject_flow
          rules:
            - pattern: '"from":\s*[1-9]\d{4,}'
              message: "Deep pagination is forbidden"
              status: 403
            - pattern: 'wildcard'
              message: "Wildcard query is forbidden"
              status: 403
            - pattern: 'regexp'
              message: "Regexp query is forbidden"
              status: 403
      - elasticsearch:
          elasticsearch: prod
          timeout: 10s

Result: dangerous DSL never reaches ES, reducing slow‑query count dramatically.

Production‑Grade Java Client Example

package com.example.search.client;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.elasticsearch.core.search.Hit;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.ElasticsearchTransport;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.message.BasicHeader;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import javax.net.ssl.SSLContext;
import java.util.List;

public class GatewayEsClientFactory {
    public static ElasticsearchClient create(String host, int port, String username, String password, SSLContext sslContext) {
        BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
        Header[] defaultHeaders = new Header[]{new BasicHeader("X-Client-App", "order-search-service")};
        RestClientBuilder builder = RestClient.builder(new HttpHost(host, port, "https"))
                .setDefaultHeaders(defaultHeaders)
                .setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
                        .setSSLContext(sslContext)
                        .setDefaultCredentialsProvider(credentialsProvider)
                        .setMaxConnTotal(200)
                        .setMaxConnPerRoute(100))
                .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
                        .setConnectTimeout(3000)
                        .setSocketTimeout(10000));
        RestClient restClient = builder.build();
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
        return new ElasticsearchClient(transport);
    }

    public static List<String> searchTitles(ElasticsearchClient client, String keyword) throws Exception {
        SearchResponse<ArticleDocument> response = client.search(s -> s
                .index("article-*")
                .query(q -> q.match(m -> m.field("title").query(keyword)))
                .size(10), ArticleDocument.class);
        return response.hits().hits().stream()
                .map(Hit::source)
                .filter(doc -> doc != null)
                .map(ArticleDocument::getTitle)
                .toList();
    }

    public static class ArticleDocument {
        private String title;
        public String getTitle() { return title; }
        public void setTitle(String title) { this.title = title; }
    }
}

The example demonstrates explicit connection‑pool sizing, TLS + Basic Auth, and the use of a custom header that propagates through the gateway for audit purposes.

Observability & Auditing

Without audit logs you cannot answer who accessed which index, when, and with what latency. Recommended audit fields include timestamp, method, path, client IP, username, application tag, status code, latency, request/response sizes, and target index.

flow:
  - name: main_flow
    filter:
      - basic_auth:
          valid_users:
            admin: ${ADMIN_PASS}
      - logging:
          elasticsearch: audit
          index: "gateway-access-logs"
          context: |
            {
              "timestamp": "_ctx.timestamp",
              "method": "_ctx.request.method",
              "path": "_ctx.request.path",
              "status": "_ctx.response.status",
              "latency_ms": "_ctx.response.took_in_millis",
              "source_ip": "_ctx.request.client_ip",
              "user": "_ctx.auth.username",
              "query_string": "_ctx.request.query_string",
              "content_length": "_ctx.request.content_length"
            }
      - elasticsearch:
          elasticsearch: prod

Four essential alert types: sudden spikes in 401/403, bulk request surges, P99 latency jumps, and high‑frequency per‑account or per‑IP access.

Performance & Capacity Planning

Gateway adds TLS handshake, header parsing, filter processing, and an extra network hop. In most production workloads the overhead is negligible compared to the stability gains, provided connection pools are tuned, keep‑alive is enabled, and rate‑limit policies keep ES from being overwhelmed.

Common Pitfalls & Troubleshooting

Using the same service account for client‑to‑gateway and gateway‑to‑ES hides the original client identity in audit logs.

Enabling node discovery without granting the gateway sufficient permissions leads to topology refresh failures.

Over‑permissive gateway limits cause ES thread‑pool exhaustion; under‑permissive limits cause 429/403 errors for legitimate traffic.

Manual certificate rotation causes downtime; automate with cert‑manager.

Exposing the management API publicly mixes control plane and data plane, weakening security boundaries.

Evolution Roadmap

Phase 1 – Quick Fix: Close public ES ports, expose only gateway HTTPS, enable basic auth.

Phase 2 – Layered Access: Introduce account tiers, path‑level rate limits, DSL risk filters, audit logging.

Phase 3 – HA & Elasticity: Deploy multiple gateway replicas, load‑balance, enable auto‑scaling, separate management API.

Phase 4 – Platform Governance: Implement read/write separation, multi‑tenant quotas, multi‑cluster traffic routing, gray‑release double‑write verification.

Final Recommendations for Architects

First, stop exposing ES directly; route all traffic through the gateway.

Second, separate client authentication from service‑account authentication.

Third, immediately rate‑limit high‑risk operations ( _bulk, deep pagination, wildcard/regexp).

Fourth, build comprehensive audit logs to enable forensic analysis and alerting.

Fifth, evolve the deployment toward a cloud‑native, multi‑replica, auto‑scaled gateway that becomes the central access‑governance hub.

When these steps are completed, the system is no longer a simple proxy but a fully‑featured, production‑grade Elasticsearch security perimeter.

References

INFINI Gateway official documentation

INFINI Gateway Helm chart

INFINI Runtime Operator