VictoriaMetrics' vlagent Hits 143k Logs/sec—How It Outperforms 8 Popular Log Collectors

Why Log Collectors Matter

In the observability triad (metrics, logs, traces), log collectors are the first step that pulls logs from container stdout, parses, tags and forwards them; their performance and correctness directly affect downstream services.

Test Design

The benchmark consists of four parts: log-generator: generates JSON logs with a strictly increasing sequence_id per pod.

Log collector under test: tails /var/log/pods or /var/log/containers and forwards via JSON Lines, Loki, Elasticsearch Bulk or OpenTelemetry protocols. log-verifier: records received sequence_id values to compute loss and end‑to‑end latency.

VictoriaMetrics + Grafana: visualises CPU, memory and other resource metrics.

All nine collectors (vlagent, Fluent Bit, Vector, OpenTelemetry Collector, Grafana Alloy, Grafana Agent, Promtail, Filebeat, Fluentd) run concurrently on a single n2-highcpu-32 VM (32 CPU, 32 GiB RAM, local SSD) inside a Kind cluster, each limited to 1 core and 1 GiB memory with default Helm chart settings and no tuning.

Throughput Results

Under a 100‑pod high‑concurrency write scenario, vlagent reaches a single‑node peak of 143 000 lines/s , which is 4.5× the best of Fluent Bit (31 300 lines/s) and 28× the best of Fluentd (5 100 lines/s). Its throughput grows almost linearly with load, whereas the other collectors hit a hard ceiling between 5 k and 40 k lines/s.

vlagent: 143 000 lines/s (baseline)

Fluent Bit: 31 300 lines/s (4.5× slower)

Vector: 25 000 lines/s (5.7× slower)

OpenTelemetry Collector: 20 500 lines/s (6.9× slower)

Grafana Alloy: 15 700 lines/s (9.1× slower)

Grafana Agent: 14 800 lines/s (9.7× slower)

Promtail: 13 400 lines/s (10.6× slower)

Filebeat: 5 250 lines/s (27.2× slower)

Fluentd: 5 100 lines/s (28× slower)

Resource Efficiency at 10k lines/s

When all collectors handle a steady 10 000 lines/s (2 pods × 5 000 lines/s), vlagent consumes only 0.062 CPU cores and 27.9 MiB memory, the lowest among all. Fluent Bit and Filebeat exceed the 1 GiB memory limit and are killed by the OOM killer.

CPU usage (cores) : vlagent 0.062, Fluent Bit 0.260, Vector 0.412, OpenTelemetry 0.491, Grafana Agent 0.552, Grafana Alloy 0.578, Promtail 0.655.

Memory usage (MiB) : vlagent 27.9, Promtail 63.0, Grafana Alloy 66.4, Grafana Agent 72.5, Fluent Bit 78.1, OpenTelemetry 106.8, Vector 153.5.

Correctness Issues Discovered

The benchmark also uncovered hidden bugs that do not appear in pure throughput tests:

Log‑rotation truncation : Fluent Bit and Vector split a single log line across two rotated files, forwarding the fragments as separate records (34 broken records for Fluent Bit, 2 for Vector) during a 1‑hour, 10 k lines/s run.

Vector silent loss : The default glob_minimum_cooldown_ms of 60 s means new pod logs can be missed for up to a minute; reducing it to 10 s eliminates the issue.

File descriptor leak : Under high load Vector opens log files faster than it processes them, causing FD count to grow and potentially filling the node’s disk.

Pod‑metadata loss : If a pod is deleted while Vector still holds its logs, the pod’s labels and annotations are lost, breaking downstream filtering that relies on that metadata.

These findings illustrate that default configurations are rarely tuned for high‑load scenarios and can introduce data‑loss risks.

vlagent Limitations

vlagent currently does not support multi‑line log merging (e.g., Java stack traces) and lacks custom parsing for non‑JSON logs such as Nginx access logs. The team plans to add these features via the LogsQL language.

Practical Recommendations for K8s Engineers

Do not trust default configurations; verify parameters like Vector’s glob_minimum_cooldown_ms for your load.

Allocate spare CPU and memory to the collector so it can handle traffic spikes.

Apply rate‑limiting or sampling at the source to protect the pipeline from log storms.

Increase the size of log files before rotation to reduce the chance of truncation.

If you need strict zero‑loss JSON logs and can tolerate the current feature set, vlagent is a solid choice; otherwise stick with a more mature collector.

Conclusion

Log collectors have been an overlooked component in observability stacks. This open‑source benchmark demonstrates that performance gaps can span orders of magnitude and that correctness bugs are often more insidious than raw throughput differences. Teams should benchmark collectors in their own environments before committing to production.