How to Log API Requests Without Slowing Down Your Server

When you run an API—whether a micro‑service that provides part of a large system or a monolithic application—logging acts as your black‑box recorder.

Each incoming request tells a story:

Who initiated the request?

Which endpoint did they hit?

How long did the response take?

Did we return the correct data or crash with a 500 error?

These answers are crucial for debugging, auditing, analysis, and compliance.

But there is a problem: if you log each API request the wrong way, you may unintentionally slow your server—sometimes dramatically. Logging can block the event loop, fill disk I/O, or even cause outages during traffic spikes.

Why logging hurts performance (if done wrong)

Most developers start with something like:

app.use((req, res, next) => {
  console.log(`${req.method}${req.url}`);
  next();
});

It is fast and effective… until you start receiving traffic.

When the problem appears:

Synchronous logging (e.g., console.log) blocks the event loop, especially when writing to files.

Disk I/O becomes a bottleneck when you write logs directly to local files.

Network latency shows up if you send logs to a remote server.

Log volume can grow beyond expectations, consuming disk space or hitting size limits.

Milliseconds per request may seem trivial—until you multiply them by thousands of concurrent requests. Suddenly your API spends more time logging than handling requests.

Golden rule for fast logging

“Never let the main request handler wait for log writes to finish.”

This means your logging system should work asynchronously and, preferably, outside the main execution path .

Step 1: Decide what to log

Before optimizing, determine what you really need to capture . Over‑logging wastes resources.

Typical API call log fields include:

Timestamp

HTTP method (GET, POST, PUT, DELETE…)

URL / route

Query parameters and body (mask sensitive data)

Client IP address

User‑agent

Response status code

Response time (latency)

Error message (if any)

Authenticated user ID (if applicable)

Request ID / correlation ID (for cross‑service tracing)

Example log entry:

{
  "time": "2025-08-11T14:23:45.123Z",
  "method": "POST",
  "url": "/api/orders",
  "status": 201,
  "responseTimeMs": 123,
  "userId": "usr_1a2b3c",
  "ip": "203.0.113.42",
  "userAgent": "Mozilla/5.0 (Macintosh...)",
  "requestId": "req_abcd1234"
}

Step 2: Use a non‑blocking logger

Never use console.log in production. Choose a logger designed for high performance and asynchronous writes.

Pino – ultra‑fast JSON logger (≈10× faster than winston)

Bunyan – structured logging with streams

Winston – flexible, multi‑transport, enterprise‑ready

Log4js – feature‑rich, inspired by Java’s Log4j

Example using Pino:

const pino = require('pino');
const logger = pino({
  level: 'info',
  transport: { target: 'pino‑pretty' }
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    logger.info({
      method: req.method,
      url: req.originalUrl,
      status: res.statusCode,
      responseTime: Date.now() - start
    });
  });
  next();
});

Why choose Pino? It is default async , uses fast JSON serialization , and is built for high throughput .

Step 3: Write logs to a buffer instead of directly

If you write each log line straight to disk or a remote server, you create latency. Instead:

Buffer logs in memory (e.g., store them in an array for a few seconds)

Flush periodically or when the buffer reaches a size limit

This turns thousands of tiny writes into fewer large writes.

Example:

let logBuffer = [];
const BUFFER_SIZE = 50;
const FLUSH_INTERVAL = 5000;

function flushLogs() {
  if (logBuffer.length > 0) {
    logger.info({ batch: logBuffer });
    logBuffer = [];
  }
}

setInterval(flushLogs, FLUSH_INTERVAL);

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    logBuffer.push({
      method: req.method,
      url: req.originalUrl,
      status: res.statusCode,
      time: Date.now(),
      latency: Date.now() - start
    });
    if (logBuffer.length >= BUFFER_SIZE) {
      flushLogs();
    }
  });
  next();
});

Step 4: Offload logging to worker processes

For high‑traffic APIs, even in‑process buffering can become a bottleneck. A better approach is to push logs to a separate logging service or background worker.

Send logs to a dedicated log service or background worker

Main API process → push log data to a message queue (RabbitMQ, Kafka, Redis Streams)

Worker consumes logs and writes them asynchronously to storage

Example using Redis Pub/Sub:

const Redis = require('ioredis');
const pub = new Redis();

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    pub.publish('api_logs', JSON.stringify({
      method: req.method,
      url: req.originalUrl,
      status: res.statusCode,
      latency: Date.now() - start
    }));
  });
  next();
});

Worker:

const sub = new Redis();
sub.subscribe('api_logs');
sub.on('message', (channel, message) => {
  const logData = JSON.parse(message);
  // Write logData to a database or log store
});

Step 5: Use asynchronous remote logging

If you must send logs to a centralized system, batch or send them asynchronously to avoid blocking requests. Common destinations include:

ELK stack (Elasticsearch, Logstash, Kibana)

Datadog

Graylog

Loggly

AWS CloudWatch

Example with Winston + CloudWatch:

const winston = require('winston');
require('winston-cloudwatch');

winston.add(new winston.transports.CloudWatch({
  logGroupName: 'my-api-logs',
  logStreamName: 'production',
  awsRegion: 'us-east-1',
  jsonMessage: true
}));

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    winston.info({
      method: req.method,
      url: req.originalUrl,
      status: res.statusCode,
      latency: Date.now() - start
    });
  });
  next();
});

Step 6: Mask sensitive data

When logging, you must protect user privacy and security.

Never log:

Passwords

Credit‑card numbers

API keys / tokens

Unencrypted personally identifiable information (PII)

Masking example:

function maskSensitive(obj) {
  const clone = { ...obj };
  if (clone.password) clone.password = '******';
  if (clone.cardNumber) clone.cardNumber = '**** **** **** ' + clone.cardNumber.slice(-4);
  return clone;
}

app.use(express.json());
app.use((req, res, next) => {
  req.body = maskSensitive(req.body);
  next();
});

Step 7: Measure and tune logging overhead

If you don’t measure, you won’t know whether logging is slowing you down.

Track these metrics:

Average response time before and after logging

CPU usage during peak traffic

Memory usage (buffers should not grow without bound)

Disk I/O statistics

Tools: ab (ApacheBench) autocannon (Node.js load testing)

APM tools (New Relic, Datadog, AppDynamics)

Step 8: Rotate and archive logs

If you log every API call, logs grow quickly.

Set up log rotation :

Rotate daily or when a file reaches a size limit (e.g., 100 MB)

Compress old logs

Move them to cheaper storage (AWS S3, Glacier)

Example with Pino:

pino server.js | tee >(pino‑pretty) | rotatelogs ./logs/api-%Y-%m-%d.log 86400

Step 9: Go distributed at scale

At very high scale:

Avoid writing logs locally; use remote collectors.

Use log shippers such as Filebeat or Fluent Bit to forward logs.

Consider Kafka + ELK to handle millions of log events per minute.

If full detail isn’t needed for every request, apply sampling (e.g., log 1 % of successful 200 responses but 100 % of 500 errors).

Step 10: A real‑world production setup

This is what a high‑scale, low‑latency API logging pipeline looks like:

API gateway (Nginx, Kong) records request metadata → sends to Kafka.

Application servers log rich information (user ID, business data) → asynchronously push to Kafka.

Kafka consumers process logs → write to Elasticsearch for search.

Kibana / Grafana dashboards visualize logs in real time.

Alerts trigger on error spikes, slow endpoints, or abnormal traffic patterns.

This setup ensures that even during traffic surges, your API is not blocked by logging overhead.

Key takeaways

Never block the request path – log asynchronously.

Use a fast logger such as Node.js’s Pino.

Buffer and batch logs before writing or sending.

Offload logging to worker processes or remote queues.

Mask sensitive data before storage.

Rotate logs and archive old data to save storage.

Monitor performance to ensure logging never becomes a bottleneck.