Why You Shouldn’t Hand Out Kibana Permissions Lightly

Incident: a query bomb crashes the cluster

An operations colleague reported an ES node failure. Monitoring showed JVM heap soaring, system load approaching 500, and I/O metrics exploding. The root cause was a single user opening Kibana, clicking “past 30 days” with no additional query conditions, which triggered a full‑cluster scan and brought the node down.

Why unrestricted Kibana access is dangerous

1. Uncontrolled user behavior

Kibana is a powerful analytics UI, but most users do not understand the data schema or query cost. They can:

Run a match_all query that scans every document;

Select a large time range such as “past 30 days”;

Submit queries without any filters.

The result is immediate CPU saturation, aggressive garbage collection, and a crashing node.

2. No built‑in query brake

With default settings, Elasticsearch executes shard queries concurrently, spawning many threads per node. The larger the time range and the more fields involved, the higher the aggregation overhead. There is no enforced limit on time range, aggregation fields, or data volume, so the system does not stop a user‑initiated overload.

3. Security risk

Granting Kibana access is equivalent to handing users a “back‑door key”. They can view index names, field mappings, and even craft custom DSL queries that bypass application logic, potentially exposing sensitive data or performing illegal scans.

Three‑layer safety rope

✅ Layer 1 – Restrict permissions

Configure Kibana Spaces to isolate different workspaces.

Create role‑based access controls so only experienced operators or analysts can use Kibana.

Assign users to the appropriate space and role.

✅ Layer 2 – Front‑end query encapsulation

Users submit queries through a business UI, not directly via Kibana.

The front‑end adds a default time range and mandatory filters.

The back‑end generates the final DSL request and sends it to Elasticsearch, preventing users from “free‑wheeling”.

Replace user‑written DSL with controlled API calls.

✅ Layer 3 – Query templates and concurrency controls

Provide preset search templates (e.g., only allow the last 7 days, single‑field aggregation).

Limit concurrent shard requests:

POST .kibana_8.15.0/_search?max_concurrent_shard_requests=2
{
  "query": { "match_all": {} }
}

Recommended value: 2–3.

Cap aggregation buckets:

PUT _cluster/settings
{
  "transient": { "search.max_buckets": 20000 }
}

Restrict scroll contexts:

PUT /_cluster/settings
{
  "persistent": { "search.max_open_scroll_context": 1000 }
}

Enforce query templates that always include a time range (mustache example omitted for brevity).

Configure thread‑pool queue size to protect the search thread pool:

threadpool.search.queue_size: 500

Enable request breaker to limit memory per request:

PUT /_cluster/settings
{
  "persistent": { "indices.breaker.request.limit": "50%" }
}

Optimization checklist

Set max_concurrent_shard_requests to 2–3.

Limit search.max_buckets to ≤ 10 000.

Configure search.max_open_scroll_context to a safe upper bound.

Apply fine‑grained Kibana permission groups.

Provide time‑range‑restricted query templates.

Block direct DSL entry; expose only encapsulated API endpoints.

Set threadpool.search.queue_size appropriately.

Enable indices.breaker.request.limit (e.g., 60%).

Conclusion

Elasticsearch is a powerful engine, but exposing it as a “BI tool” to end users without safeguards is unsafe. Controlling entry points, enforcing permission tiers, and guiding user behavior are essential to keep the cluster stable and secure.