Why OpenTelemetry Helm Splits into 3 Releases: Agent, Cluster, Gateway Architecture Explained

1. Mechanism Leap: How Presets Ended "Configuration Hell"

To understand the architectural split, first grasp the most important recent configuration feature in the OpenTelemetry Helm Chart: Presets .

In early deployment modes, configuring a Collector to collect host and container metrics was extremely painful:

Receiver and pipeline assembly : You had to manually define dozens of hostmetrics scrapers (cpu, memory, disk, filesystem, load, etc.) and configure authentication service accounts and insecure-skip parameters for kubeletstats.

Error-prone host volume mounts : Containers cannot read physical node hardware metrics by default, so you had to manually declare hostPath mounts for host /proc, /sys, /var/run/docker.sock directories. A missing or wrong mount caused silent metric loss.

RBAC permission bloat and drift : Collecting pod status required pods permissions, nodes required nodes/proxy, events required events. Manually writing ClusterRole rules was tedious and prone to permission gaps during OTel Collector version upgrades.

To solve this, the official Helm Chart introduced the out-of-the-box Presets engine :

Via Presets, hundreds of lines of low-level resource assembly were abstracted into minimal boolean declarations:

presets:
  # 1. Auto-mount /proc and /sys, assemble hostmetrics receiver
  hostMetrics:
    enabled: true
  # 2. Auto-grant nodes/proxy permission, assemble kubeletstats receiver
  kubeletMetrics:
    enabled: true
  # 3. Auto-inject k8sattributes processor, extract pod metadata labels
  kubernetesAttributes:
    enabled: true
  # 4. Auto-grant cluster read-only permissions, assemble k8s_cluster receiver
  clusterMetrics:
    enabled: true

When you set a Preset to enabled: true, the Helm template engine automatically performs a trinity injection behind the scenes:

Auto-configure components and pipelines : Inject the corresponding Receiver or Processor into the Collector runtime's service.pipelines.

Auto-mount required volumes : Inject /proc, /sys, or /var/log/pods mounts into the pod template as needed; unused Presets never mount extra host directories.

Auto-generate least-privilege RBAC : Precisely grant only the API groups and resource verbs required for the current function in the generated ClusterRole, fully compliant with least-privilege security baselines.

However, Presets solved "how to configure" but raised a more severe architectural question: "where to run"? If you enable all four Presets in a single Chart Release, whether you run as DaemonSet or Deployment, you will directly trigger production disasters.

2. Architecture Breakdown: Three Releases' Responsibilities and Lifecycle Isolation

This is why the community and upstream must split deployment into three Releases. They appear to use the same Helm Chart ( open-telemetry/opentelemetry-collector), but behind them lie completely different compute modes, security permissions, blast radii, and scaling dimensions .

Release 1: otel-agent (Node-side Daemon)

Deployment mode : mode: daemonset Replica count : Strictly 1:1 with Kubernetes physical worker nodes

Enabled Presets : hostMetrics, kubeletMetrics, kubernetesAttributes (optionally logsCollection)

Core Responsibilities and Design Constraints

otel-agent

is the infrastructure probe residing on each compute node. Its core task is to scrape the host's own hardware utilization (CPU, memory, network, IO) and the Kubelet container metrics running on the current node.

Strong host binding : Because hostMetrics is enabled, the Agent container must mount host /proc and /sys, and obtain its node name via the KUBE_NODE_NAME environment variable, communicating directly with the local Kubelet via localhost or node-internal network (port 10250).

Blast radius limitation : If an Agent on a machine suffers memory leak, gets OOM-killed, or the host has physical failure, the loss is only that single machine's basic monitoring; all other nodes' business traces and metrics remain completely unaffected.

Fixed lifecycle : Its scaling changes only with worker node addition/removal; it absolutely cannot and does not need dynamic HPA scaling based on business HTTP/gRPC request traffic.

Release 2: otel-cluster (Control-plane Global Singleton)

Deployment mode : mode: deployment Replica count : Strictly 1 (Singleton)

Enabled Presets : clusterMetrics (based on k8s_cluster receiver), optionally kubernetesEvents (based on k8sobjects receiver)

Why Must It Be a Strict Singleton (replicaCount: 1)?

This is the most confusing point for newcomers: cloud-native architecture emphasizes high availability, so why can otel-cluster only have 1 replica?

The k8s_cluster receiver works by calling the Kubernetes API Server to pull cluster-wide object metadata and scheduling state (e.g., desired vs ready replicas of Deployments per namespace, Node condition statuses, distribution of Pods in Pending/CrashLoopBackOff, etc.).

Eliminate data duplication pollution : The Kubernetes API Server returns a global state view. If you casually set otel-cluster to 3 replicas, 3 Pods would each independently poll the API Server and push 3 identical copies of time-series data to the monitoring backend! On Grafana dashboards, your cluster pod count and resource consumption would instantly triple.

Protect Kube-APIServer from avalanche impact : Large clusters contain hundreds of thousands of K8s resource objects. One singleton pod polling periodically suffices for metrics; multiple replicas would only multiply List/Watch pressure on the API Server.

Decouple scaling dimension : otel-cluster 's resource overhead scales linearly only with the number of Kubernetes resource objects in the cluster , and has no direct relationship with user business request concurrency (QPS). Isolating it allows allocating very small, fixed CPU/memory limits (e.g., 0.2 core, 256MB) for rock-solid operation.

Release 3: otel-gateway (Data-plane Elastic Central Gateway)

Deployment mode : mode: deployment Replica count : Default 2+, with HPA (Horizontal Pod Autoscaler) elastic scaling

Enabled Presets : All disabled (no infrastructure Presets needed)

Core Responsibilities and Governance Capabilities

otel-gateway

is the data hub of the entire observability pipeline. All business microservices (via OpenTelemetry SDK) send their Traces, Metrics, and Logs directly to the Gateway's unified Service endpoints (gRPC: 4317, HTTP: 4318).

Privilege-free and zero host dependency : Gateway is a pure network proxy stateless service. It needs no /proc mounts, no Kubelet access, no special host privileges—extremely high security.

Heavy computation and intelligent governance :

Memory circuit breaker ( memory_limiter ) : Proactively drops low-priority telemetry or triggers backpressure before burst traffic crushes the Collector, preventing Pod OOM.

Tail-based smart sampling ( tail_sampling ) : For microservice full-link tracing, 99% of 200-success requests don't need full ingestion. Gateway can judge by status code and latency, retain 100% of error traces and slow queries, drastically cutting backend storage costs.

Multi-backend routing and redaction (Routing & Redaction) : Scrub sensitive attributes (ID numbers, passwords) from traces, export Traces to Jaeger/Tempo, push business metrics to Prometheus/VictoriaMetrics.

Pure business-traffic-driven HPA : Since it binds neither to hosts nor API Server, Gateway can scale from 2 replicas to 20 or even 50 within seconds based on its own CPU consumption or received span rate, perfectly absorbing Double-11 or promotional traffic peaks.

3. Architectural Conflicts: Why Can't They Merge into a Monolith?

Understanding the three components' division of labor, let's reason: if forced into one Release, what fatal conflicts arise?

Conflict 1: DaemonSet Cannot Serve as Elastic Data Gateway

If each machine's otel-agent also acts as Gateway receiving full business traces and doing tail sampling:

Memory explosion : Tail sampling requires in-memory sliding time windows to buffer all spans of the same TraceId. Placing this memory overhead in DaemonSet would cause massive Agent OOM on worker nodes during traffic peaks, even triggering node-level resource eviction.

Cross-node trace fragmentation : Microservice requests flow across machines; spans of the same TraceId land on different nodes. Dispersed DaemonSets cannot perceive the global complete trace, making cross-microservice global sampling decisions difficult.

Conflict 2: Gateway Must Not Mix In clusterMetrics

If you put clusterMetrics: enabled: true into otel-gateway for convenience:

During business lows, Gateway has 2 replicas; during promotional peaks, HPA scales Gateway to 30 replicas. At that point, 30 Collector instances would simultaneously high-frequency poll Kube-APIServer ! The cluster control plane could be instantly paralyzed by its own monitoring component.

It is precisely this fundamental contradiction in lifecycle and scaling dimensions among hardware binding (Agent), control-plane global (Cluster), and data-plane high concurrency (Gateway) that drives the upstream recommended topology toward decoupling.

4. Selection Trade-offs: When Can You Converge to "Two Releases"?

Although the official gold standard is three Releases, the user question accurately mentions "or two." In real production, converging to two Releases is indeed reasonable in certain scenarios.

Option A: Converge to otel-agent + otel-gateway (Merge Cluster)

This is currently the most common dual-Release variant:

Approach : Enable presets.clusterMetrics.enabled: true in otel-gateway 's Deployment, but must explicitly enable Leader Election .

Mechanism : Although Gateway runs multiple replicas, only the Leader pod that acquires the Lease lock polls the API Server; other Standby pods serve only as ordinary OTLP traffic gateways.

Applicable boundary : Small clusters (worker nodes < 50), few gateway replicas (2-4), and non-drastic HPA changes.

Potential risk : When Gateway triggers rolling updates or pod scaling due to business traffic peaks, Leader switching may cause brief control-plane metric gaps.

Option B: Converge to otel-agent + otel-cluster (No-Gateway Direct Mode)

Approach : Business pods obtain current node's status.hostIP via Kubernetes Downward API, push OTLP telemetry directly to the node's otel-agent; Agent batches and pushes data to external managed monitoring platforms (e.g., cloud vendor managed Prometheus/Grafana/Datadog).

Applicable boundary : Team doesn't need self-built complex tail-sampling pipelines, observability backend fully outsourced to cloud vendor, pursuing minimal component maintenance.

5. Production Landing: A Clear Helm Values Checklist

To land this standard architecture in a production cluster, you only need to maintain three responsibility-clear values files and deploy three times based on the same official Chart:

# Add official Helm repo
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

1. values-gateway.yaml (Deploy Gateway First)

mode: deployment

replicaCount: 2

# Production must configure resource constraints and memory limits
resources:
  limits:
    cpu: "2"
    memory: 2Gi
  requests:
    cpu: 500m
    memory: 512Mi

# Enable Gateway's own HPA
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

# Pure gateway mode, disable all infrastructure Presets
presets:
  hostMetrics:
    enabled: false
  kubeletMetrics:
    enabled: false
  clusterMetrics:
    enabled: false
  kubernetesAttributes:
    enabled: false

# Core pipeline: OTLP receive → memory limiter & batch → export storage
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 20
    batch:
      send_batch_size: 1024
      timeout: 1s

  exporters:
    # Example: traces to Jaeger, metrics to Prometheus
    otlp/jaeger:
      endpoint: jaeger-collector.observability.svc:4317
      tls:
        insecure: true

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/jaeger]

2. values-agent.yaml (Deploy Node Agent)

mode: daemonset

presets:
  hostMetrics:
    enabled: true
  kubeletMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
  logsCollection:
    enabled: false # Enable on demand for node container log collection

# Push node metrics via internal network to Gateway
config:
  exporters:
    otlp/gateway:
      endpoint: otel-gateway.observability.svc:4317
      tls:
        insecure: true

  service:
    pipelines:
      metrics:
        # hostmetrics and kubeletstats are auto-composed by Preset and added to receivers
        processors: [k8sattributes, batch]
        exporters: [otlp/gateway]

3. values-cluster.yaml (Deploy Control-plane Singleton)

mode: deployment

# Iron law: strict singleton
replicaCount: 1

presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true

config:
  exporters:
    otlp/gateway:
      endpoint: otel-gateway.observability.svc:4317
      tls:
        insecure: true

  service:
    pipelines:
      metrics:
        # k8s_cluster is auto-injected by Preset
        processors: [batch]
        exporters: [otlp/gateway]

Sequential Installation Commands

# 1. Install gateway data hub
helm upgrade --install otel-gateway open-telemetry/opentelemetry-collector \
  -n observability --create-namespace -f values-gateway.yaml

# 2. Install control-plane singleton collector
helm upgrade --install otel-cluster open-telemetry/opentelemetry-collector \
  -n observability -f values-cluster.yaml

# 3. Install node daemon
helm upgrade --install otel-agent open-telemetry/opentelemetry-collector \
  -n observability -f values-agent.yaml

Summary: From "Monolithic Toy" to "Industrial Division of Labor"

Reviewing this evolution of the OpenTelemetry Helm Chart, it is essentially the inevitable result of cloud-native monitoring systems moving from "lab prototype" to "ultra-large-scale production environments" :

Configuration level : Through Presets , the underlying complex assembly (Receivers + VolumeMounts + RBAC) is encapsulated into high-cohesion business intent declarations, ending hundreds of lines of configuration hell.

Topology level : By splitting the Chart into otel-agent, otel-cluster, otel-gateway three independent Releases, it thoroughly achieves separation of concerns among host privileges, control-plane state, and data-plane concurrency .

For medium-to-large clusters, the three-Release topology keeps your observability pipeline composed and stable through extreme traffic bursts, node outages, and rolling updates. Next time you plan observability in Kubernetes, consider bidding farewell to that bulky monolithic YAML and embracing this clearly layered industrial-grade topology.