Ingest Metrics, Traces, Alerts into OpenObserve with Prometheus & OpenTelemetry

Metrics

OpenObserve supports metric ingestion via Prometheus remote_write. Use node_exporter to collect node metrics and configure Prometheus to remote write to OpenObserve.

# prometheus.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: openobserve
data:
  prometheus.yaml: |
    global:
      scrape_interval: 15s
      scrape_timeout: 15s
    remote_write:
    - url: http://openobserve.openobserve.svc.cluster.local:5080/api/default/prometheus/api/v1/write
      basic_auth:
        username: [email protected]
        password: root321
    scrape_configs:
    - job_name: "nodes"
      static_configs:
      - targets: ['10.206.16.6:9100', '10.206.16.5:9100', '10.206.16.10:9100']
      relabel_configs:
      - source_labels: [__address__]
        regex: "(.*):(.*)"
        replacement: "${1}"
        target_label: 'ip'
        action: replace
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: openobserve
spec:
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - args:
        - --config.file=/etc/prometheus/prometheus.yaml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention.time=4h
        - --web.enable-lifecycle
        image: prom/prometheus:v2.44.0
        imagePullPolicy: IfNotPresent
        name: prometheus
        ports:
        - containerPort: 9090
          name: http
          protocol: TCP
        securityContext:
          runAsUser: 0
        volumeMounts:
        - mountPath: /etc/prometheus
          name: config-volume
        - mountPath: /prometheus
          name: data
      volumes:
        - name: data
          emptyDir: {}
        - configMap:
            defaultMode: 420
            name: prometheus-config
          name: config-volume
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: openobserve
spec:
  ports:
  - name: http
    port: 9090
    targetPort: 9090
  selector:
    app: prometheus
  type: NodePort

Apply the manifest with kubectl apply -f prometheus.yaml and verify pods and services. The Prometheus UI shows metrics, which are also visible in OpenObserve's UI as streams (e.g., node_load5).

OpenObserve currently supports limited PromQL features; upcoming releases will add full vector operations.

Metrics can also be queried using SQL syntax.

Tracing

OpenObserve supports OpenTelemetry‑based tracing. Use the OpenTelemetry SDK or Collector to send trace data.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

resource = Resource(attributes={SERVICE_NAME: "python-service"})
tracer_provider = TracerProvider(resource=resource)
url = 'HTTP_Endpoint'
headers = {"Authorization": "Authorization"}
exporter = OTLPSpanExporter(endpoint=url, headers=headers)
span_processor = BatchSpanProcessor(exporter)
tracer_provider.add_span_processor(span_processor)
trace.set_tracer_provider(tracer_provider)

Replace HTTP_Endpoint and Authorization with your OpenObserve endpoint (e.g., https://url:5080/api/<orgname>/traces) and a Basic auth header.

# OpenTelemetry Collector configuration (excerpt)
exporters:
  otlphttp:
    traces_endpoint: http://url:5080/api/<orgname>/traces
    headers:
      Authorization: Basic base64(userid:password)
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]
      processors: [batch]

Traces appear in OpenObserve's UI, where you can inspect individual trace details.

Alerting

OpenObserve provides scheduled and real‑time alerts. Create an alert template (e.g., for Slack) with placeholders like {stream_name}, {org_name}, {alert_name}, and {alert_type}.

{
  "text": "For stream {stream_name} of organization {org_name} alert {alert_name} of type {alert_type} is active"
}

Define a destination (e.g., Slack webhook) and then create an alert. Example: a scheduled alert triggers when the K8sLogs stream receives more than 50 logs in one minute, sending a notification to Slack.

select count(*) as echocnt FROM 'K8sLogs' WHERE (_timestamp >= 1691488182902275 AND _timestamp < 1691488242902275) LIMIT 100

Real‑time alerts evaluate conditions during ingestion.

OpenObserve also supports Vector Remap Language (VRL) for advanced data processing during ingestion or query.