How to Build Unified Observability for Apache APISIX with DeepFlow

Installation

Deploy DeepFlow and Apache APISIX as services in a single all‑in‑one Kubernetes cluster. The deployment follows the official DeepFlow All‑in‑One installation guide and the APISIX installation guide. The Linux kernel on the host must be version 4.14 or newer to enable DeepFlow’s eBPF observability.

Distributed Tracing

DeepFlow supports two tracing approaches for APISIX and its backend services:

Method 1 – DeepFlow eBPF AutoTracing : No APISIX plugins are required. Install deepflow-agent on the APISIX node and use the DeepFlow Distributed Tracing Dashboard in Grafana to start a trace and view the full‑stack call chain across APISIX and the backends.

Method 2 – DeepFlow eBPF + OpenTelemetry : When backend services already emit OpenTelemetry traces, enable the APISIX opentelemetry plugin, send the trace data to DeepFlow, and obtain a combined flame‑graph that includes application SPANs (generated by the APM), system SPANs (eBPF) and network SPANs (cBPF).

Demo of Method 2 uses a Spring Boot “WebShop” micro‑service with OpenTelemetry support. The APISIX route and plugin configuration are:

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: deepflow-apisix-demo
  namespace: deepflow-otel-spring-demo
spec:
  http:
    - name: deepflow-apisix-demo
      match:
        hosts:
          - apisix.deepflow.demo
        paths:
          - "/*"
      backends:
        - serviceName: web-shop
          servicePort: 18090

plugins:
  - opentelemetry
pluginAttrs:
  opentelemetry:
    resource:
      service.name: APISIX
    collector:
      address: deepflow-agent.deepflow.svc.cluster.local/api/v1/otel/trace
      request_timeout: 3

After enabling the plugin via the APISIX admin API, a request such as

curl -H "Host: apisix.deepflow.demo" 10.1.23.200:44640/shop/full-test

appears in the DeepFlow Distributed Tracing Dashboard, showing a flame‑graph where APM‑generated application SPANs (A) are linked with DeepFlow‑generated system (S) and network (N) SPANs.

Performance Metrics

DeepFlow provides out‑of‑the‑box RED (Rate, Errors, Duration) metrics at the endpoint level and detailed TCP metrics. APISIX’s Metrics plugins (e.g., Prometheus, node‑status) expose HTTP status codes, bandwidth, connection counts, and latency. These metrics can be integrated into DeepFlow and visualized in APISIX‑provided Grafana dashboards.

Enable the Prometheus plugin in APISIX by adding it to .values.yaml:

plugins:
  - prometheus
pluginAttrs:
  prometheus:
    export_uri: /metrics
    export_addr:
      ip: 0.0.0.0
      port: 9091
    metrics:
      http_status:
        extra_labels:
          - upstream_addr: $upstream_addr
          - upstream_status: $upstream_status

Expose the metrics via a ServiceMonitor CRD (Prometheus operator) and configure RemoteWrite to forward them to DeepFlow:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  namespace: monitoring
spec:
  enableRemoteWriteReceiver: true
  remoteWrite:
    - url: "http://deepflow-agent.deepflow.svc.cluster.local/api/v1/prometheus"

DeepFlow’s agent receives Prometheus data by default; no additional configuration is required.

Access Logs and Continuous Profiling

Deploy deepflow-agent on the APISIX node. The DeepFlow Application‑Request Log Dashboard in Grafana automatically displays request/response headers, latency, and error codes without any APISIX configuration changes.

In the enterprise edition, DeepFlow also captures on‑CPU profiles and function‑call stacks via eBPF, showing time spent in business functions, shared libraries, and kernel syscalls.

Apache APISIX

Apache APISIX is a dynamic, real‑time, high‑performance open‑source API gateway built on NGINX and LuaJIT. It supports load balancing, dynamic upstreams, canary releases, circuit breaking, authentication, and observability. Single‑core QPS can reach 23 000 with an average latency of 0.2 ms. GitHub repository: https://github.com/apache/apisix

DeepFlow

DeepFlow, developed by Cloudsprout Network, is an observability platform that provides zero‑code, full‑stack data collection for cloud‑native applications using eBPF. It captures application performance metrics, distributed tracing, and continuous profiling, and links all signals through SmartEncoding tags. GitHub repository: https://github.com/deepflowio/deepflow

References

DeepFlow All‑in‑One installation guide: https://deepflow.io/docs/zh/ce-install/all-in-one/

APISIX installation guide: https://apisix.apache.org/docs/apisix/3.2/installation-guide/

DeepFlow Demo – Spring Boot WebShop: https://deepflow.io/docs/zh/integration/input/tracing/opentelemetry/#基于-spring-boot-demo-体验

DeepFlow metrics documentation: https://deepflow.io/docs/zh/features/universal-map/metrics-and-operators/

APISIX Grafana dashboard JSON: https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json

APISIX Prometheus plugin docs: https://apisix.apache.org/docs/apisix/plugins/prometheus/

DeepFlow demo overview: https://deepflow.io/docs/zh/ce-install/overview/