How to Benchmark Sentry SDK Overhead on Web Servers

Motivation

Developers often cannot answer how much overhead a monitoring SDK adds, nor what level of overhead is acceptable. This article focuses on measuring that impact for web‑server performance monitoring.

How It Works

The sentry-sdk-benchmark repository provides a set of pre‑packaged web applications written in various languages and frameworks. For each application in the baseline directory (taken from the TechEmpower Framework Benchmarks), there is a corresponding copy in the instrumented directory that adds the minimal Sentry error‑and‑performance‑monitoring changes.

Running the benchmark runner creates one or more Docker Compose projects, launches the baseline and instrumented apps side by side, and generates an HTML report comparing latency, CPU, memory, and other metrics.

Test Setup

Each Docker Compose project starts the target app, a PostgreSQL database, and a load generator. The load generator issues a fixed‑rate “open model” request stream to the app, following the methodology described in Schroeder et al., “Closed versus open system models and their impact on performance and scheduling.”

When the app is instrumented with the Sentry SDK, the SDK is configured to send data to a Mock Sentry Ingestion Server, which mimics the real Sentry ingestion pipeline without external network calls.

Usage

Install the latest Docker (with Docker Compose V2) and Go 1.17+.

Compile the benchmark runner: go build . (Optional) Move the sentry-sdk-benchmark binary into a directory in $PATH, e.g. mv sentry-sdk-benchmark /usr/local/bin/ Run the tool for a specific platform, for example:

sentry-sdk-benchmark platform/python/django

Cleaning Up Resources

After the run the tool attempts to clean up containers, images, and networks. If leftovers remain, the following commands can be used (exercise caution as they may affect resources not created by the benchmark):

<code>docker compose ls -a -q | xargs -tI '{}' docker compose -p '{}' down --remove-orphans --rmi local</code> Delete all Docker containers: <code>docker ps -a -q | xargs -tn10 docker rm -f</code> Prune unused Docker networks: <code>docker network prune</code> Remove images labeled with io.sentry.sentry-sdk-benchmark : <code>docker images -f "label=io.sentry.sentry-sdk-benchmark" -q | sort -u | xargs -tn10 docker rmi -f</code> Delete dangling images: <code>docker images -f "dangling=true" -q | xargs -tn10 docker rmi -f</code>

Adding More Platforms

To add additional language or framework implementations, see platform/README.md in the repository.