How to Enable Distributed Tracing in NGINX with OpenTracing and Jaeger (Step‑by‑Step)

Background

NGINX is a widely used, general‑purpose application server and the most popular web server. It can serve static files, but it is also commonly used as a reverse proxy, load balancer, or API gateway in distributed systems.

Distributed tracing is a mechanism for analyzing and monitoring applications by tracking a single request across its entire path from source to destination, unlike tracing confined to a single application domain.

In other words, distributed tracing stitches together multiple requests across several systems, usually using one or more related IDs. The trace consists of a set of structured log events stored centrally.

In this context, OpenTracing was created. OpenTracing is a vendor‑agnostic API that helps developers easily trace a single request across domains. Several open‑source products (e.g., Jaeger, SkyWalking) support OpenTracing as a standardized way to collect distributed traces.

This article demonstrates, from zero to one, how to configure distributed tracing for NGINX using a simple example architecture. The components used are:

nginx[1] v1.22

jaeger‑all‑in‑one[2] v1.38

nginx‑opentracing[3] v1.22

jaeger‑client‑cpp[4] v0.9

Building nginx‑opentracing

Prepare nginx‑opentracing

The nginx‑opentracing repository provides a pre‑built dynamic library for each NGINX version (NGINX 1.19.13+). You can use the library directly, but if you try to embed it as a built‑in module with --add-module=/path/to/module, you may encounter errors such as:

ngx_http_opentracing_module.so/config was found

/root/nginx‑opentracing‑0.25.0/opentracing//src/ngx_http_opentracing_module.cpp
In file included from /root/nginx‑opentracing‑0.25.0/opentracing//src/ngx_http_opentracing_module.cpp:1:0:
/root/nginx‑opentracing‑0.25.0/opentracing//src/load_tracer.h:3:38: fatal error: opentracing/dynamic_load.h: No such file or directory

According to issue[6], nginx‑opentracing requires the opentracing‑cpp library. Because of C++ compilation difficulties, the author chose to use the official dynamic library instead of building it from source.

Prepare jaeger‑client‑cpp

nginx‑opentracing also needs a Jaeger tracer, provided by the jaeger‑client‑cpp library. The released binaries are outdated, so the latest version must be compiled manually.

Note: Compilation requires CMake 3.3+ and gcc 4.9.2+

The build environment uses CentOS 7 with the default gcc and CMake, but both gcc and CMake need to be compiled to the required versions.

Compile gcc

Download gcc:

https://ftp.gnu.org/gnu/gcc/

cd gcc-5.4.0
./contrib/download_prerequisites

mkdir gcc-build-5.4.0
cd gcc-build-5.4.0

/usr/local/src/gcc-5.4.0/configure \
  --enable-checking=release \
  --enable-languages=c,c++ \
  --disable-multilib

make && make install

Reference: Upgrade GCC[10]

cd /usr/bin/
mv gcc gcc_back
mv g++ g++_back
ln -s /usr/local/bin/gcc gcc
ln -s /usr/local/bin/g++ g++

During compilation, the following error may appear:

/lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found

The fix is to restore the original libgcc dynamic library.

configure: error: C++ compiler missing or inoperational

make[2]: *** [configure-stage1-libcpp] Error 1
make[2]: Leaving directory `/home/clay/programming/C++/gcc-4.8.1'
make[1]: *** [stage1-bubble] Error 2
make[1]: Leaving directory `/home/clay/programming/C++/gcc-4.8.1'
make: *** [all] Error 2

Compile cmake

./configure --prefix=/path/to/app

make
make install

If the custom gcc is used, set the library path:

LD_LIBRARY_PATH=/usr/local/lib64 ./configure --prefix=/usr/local/cmake

Compile jaeger‑client‑cpp

Follow the official steps:

cd jaeger-client-cpp-0.9.0/
mkdir build
cd build
# It is recommended to use a proxy for downloading dependencies
ALL_PROXY=http://x.0.0.x:10811 /usr/local/cmake/bin/cmake ..
make

Note: Dependency download may take a long time; just wait until it finishes.

After compilation, libjaegertracing.so.0.9.0 is the required library.

Compile nginx

./configure \
  --user=web_www \
  --group=web_www \
  --with-pcre \
  --with-compat \
  --with-http_ssl_module  \
  --with-http_gzip_static_module \
  --prefix=/root/nginx  \
  --with-http_stub_status_module

The --with-compat flag is required; otherwise the dynamic module will be reported as binary‑incompatible at startup:

nginx: [emerg] module "/root/nginx/conf/ngx_http_opentracing_module.so" is not binary compatible in /root/nginx/conf/nginx.conf:1

If the error "cc not found" occurs, create a symlink named cc pointing to gcc.

Configuring NGINX

Prepare Jaeger client configuration

File jaeger.json (parameters follow the Jaeger configuration documentation):

{
  "service_name": "nginx",
  "sampler": {
    "type": "const",
    "param": 1
  },
  "reporter": {
    "localAgentHostPort": "jaeger:6831"
  },
  "headers": {
    "jaegerDebugHeader": "jaeger-debug-id",
    "jaegerBaggageHeader": "jaeger-baggage",
    "traceBaggageHeaderPrefix": "uberctx-"
  },
  "baggage_restrictions": {
    "denyBaggageOnInitializationFailure": false,
    "hostPort": ""
  }
}

Enable OpenTracing in NGINX

For more OpenTracing parameters, see Reference.md[13].

# Load the OpenTracing dynamic module.
load_module conf/ngx_http_opentracing_module.so;
worker_processes  1;
user root root;

events {
    worker_connections  1024;
}
http {
    log_format opentracing '{"timestamp":"$time_iso8601","source":"$server_addr","hostname":"$hostname","ip":"$http_x_forwarded_for","traceID":"$opentracing_context_uber_trace_id","client":"$remote_addr","request_method":"$request_method","scheme":"$scheme","domain":"$server_name","referer":"$http_referer","request":"$request_uri","args":"$args","size":$body_bytes_sent,"status": $status,"responsetime":$request_time,"upstreamtime":"$upstream_response_time","upstreamaddr":"$upstream_addr","http_user_agent":"$http_user_agent","https":"$https"}';
    # Load tracer (using Jaeger) and provide the configuration file.
    opentracing_load_tracer conf/libjaegertracing.so conf/jaeger.json;
    # Enable tracing.
    opentracing on;
    # Optional tag example.
    opentracing_tag http_user_agent $http_user_agent;
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;
        location / {
            opentracing_operation_name $uri;
            opentracing_propagate_context;
            root   html;
            index  index.html index.htm;
        }
        access_log logs/access.log opentracing;
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Note: The dynamic library used is ot16 (linux‑amd64‑nginx‑1.22.0‑ot16‑ngx_http_module.so.tgz). Other versions are incompatible and will cause syntax‑check errors.

Configuration Details

Each location can have its own alias; this explains the difference between opentracing_operation_name and opentracing_location_operation_name.

http {
...    
    location = /upload/animal {
      opentracing_location_operation_name upload;
    ...

More configuration details are available in Tutorial.md[14].

After reloading NGINX, you can view the spans in Jaeger; only NGINX spans appear because no other back‑ends are configured.

References

nginx: https://nginx.org/en/download.html

jaeger‑all‑in‑one: https://www.jaegertracing.io/docs/1.35/getting-started/

nginx‑opentracing: https://github.com/opentracing-contrib/nginx-opentracing

jaeger‑client‑cpp: https://github.com/jaegertracing/jaeger-client-cpp

issue: https://github.com/opentracing-contrib/nginx-opentracing/issues/120

opentracing‑cpp: https://github.com/opentracing/opentracing-cpp

gcc download: https://ftp.gnu.org/gnu/gcc/

Upgrade GCC article: https://www.jianshu.com/p/8ce98a06492f

Dynamic library path issue: https://stackoverflow.com/questions/53592796/libstdc-so-6-version-glibcxx-3-4-20-not-found

Jaeger client configuration: https://github.com/jaegertracing/jaeger-client-cpp#configuration-via-environment

Reference.md: https://github.com/opentracing-contrib/nginx-opentracing/blob/master/doc/Reference.md

Tutorial.md: https://github.com/opentracing-contrib/nginx-opentracing/blob/master/doc/Tutorial.md