Why OpenObserve Beats Elasticsearch with 140× Lower Storage Costs

OpenObserve vs Elasticsearch

OpenObserve is an open‑source, high‑performance, cloud‑native observability platform written in Rust that handles logs, metrics, and traces. Compared with Elasticsearch, it can reduce storage costs by about 140 times by storing unindexed data in compressed local disks or Parquet files in object storage.

While full‑scan searches may be slower, partitioning, caching, and other techniques keep performance high. Aggregation queries are often much faster than Elasticsearch because of columnar storage.

Architecture

OpenObserve can run in single‑node mode or HA mode.

Single‑Node Mode

Sled + Local Disk : Simple mode for testing or low‑availability needs; can process over 2 TB per day on a Mac M2 (≈31 MB/s).

Sled + Object Storage : Stores data in object storage for higher durability.

Etcd + Object Storage : Uses Etcd for metadata while data resides in object storage.

HA Mode

In HA mode, OpenObserve runs multiple stateless nodes; data is stored in object storage and metadata in Etcd. The main components are Router, Ingester, Querier, and Compactor.

Router : Routes requests to Ingester or Querier and provides the UI.

Ingester : Receives ingestion requests, converts data to parquet, stores temporarily in a WAL, then writes to object storage.

Querier : Stateless component that handles queries.

Compactor : Merges small files into larger ones, manages retention policies, and updates file list indexes.

Installation

OpenObserve can be installed by downloading the binary or using Docker. The example below shows deployment on a Kubernetes cluster using the default Sled + local‑disk mode. $ kubectl create ns openobserve Create a manifest file (openobserve.yaml) with a Service and a StatefulSet:

# openobserve.yaml
apiVersion: v1
kind: Service
metadata:
  name: openobserve
  namespace: openobserve
spec:
  clusterIP: None
  selector:
    app: openobserve
  ports:
    - name: http
      port: 5080
      targetPort: 5080
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openobserve
  namespace: openobserve
  labels:
    app: openobserve
spec:
  serviceName: openobserve
  replicas: 1
  selector:
    matchLabels:
      app: openobserve
  template:
    metadata:
      labels:
        app: openobserve
    spec:
      securityContext:
        fsGroup: 2000
        runAsUser: 10000
        runAsGroup: 3000
        runAsNonRoot: true
      containers:
        - name: openobserve
          image: public.ecr.aws/zinclabs/openobserve:latest
          env:
            - name: ZO_ROOT_USER_EMAIL
              value: [email protected]
            - name: ZO_ROOT_USER_PASSWORD
              value: root321
            - name: ZO_DATA_DIR
              value: /data
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 4096m
              memory: 2048Mi
            requests:
              cpu: 256m
              memory: 50Mi
          ports:
            - containerPort: 5080
              name: http
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: cfsauto
        resources:
          requests:
            storage: 10Gi

Apply the manifest:

$ kubectl apply -f openobserve.yaml
$ kubectl get pods -n openobserve
$ kubectl get svc -n openobserve

Quick Start

Check the logs to confirm the service started: $ kubectl logs -f openobserve-0 -n openobserve Port‑forward the service and open the UI in a browser:

$ kubectl port-forward svc/openobserve 5080:5080 -n openobserve

Log in with the admin email and password set in the manifest.

Ingestion Example

Download sample log data and ingest it via the JSON API:

$ curl -L https://zinc-public-data.s3.us-west-2.amazonaws.com/zinc-enl/sample-k8s-logs/k8slog_json.json.zip -o k8slog_json.json.zip
$ unzip k8slog_json.json.zip
$ curl http://localhost:5080/api/default/default/_json -i -u "[email protected]:root321" -d "@k8slog_json.json"

After ingestion, the data appears in the Streams view and can be queried.

Query Examples

Full‑text search for the term error: match_all('error') Case‑insensitive search: match_all_ignore_case('error') Column‑specific search: str_match(fieldname, 'error') Exact match on a numeric field: code=200 Filter by stream name: stream='stderr' Extract IP from log and filter: extract_ip(log) | code=200 OpenObserve also supports metrics and tracing, though this guide focuses on log collection.

Reference documentation: https://openobserve.ai/docs