Why VictoriaLogs Is Fast: Architecture-First Performance Engineering in Go

Log systems are among the most underestimated performance challenges in infrastructure software. Handling hundreds of thousands of log entries per second with evolving fields, massive historical queries, and constrained storage costs requires more than just Go's goroutines, channels, and garbage collector.

A Counterintuitive Conclusion: Performance Optimization Starts with Doing Less Work

VictoriaMetrics principal engineer Jesús Espino emphasizes that system performance depends primarily on architecture and data structures; micro-optimizations are secondary. Consider searching 100 million logs for "timeout". Approach A optimizes per-record processing (faster string matching, fewer allocations) but still touches all 100 million records. Approach B uses indexes and filters to skip irrelevant data blocks entirely, potentially reading only a tiny fraction. The article argues for prioritizing Approach B.

Layer 1: Data Architecture Sets the Performance Ceiling

Why Log Databases Need LSM-Trees

VictoriaLogs uses an LSM-tree (Log-Structured Merge-tree). New logs enter an in-memory buffer, then are flushed to immutable disk segments. Background merges combine smaller segments into larger ones. This defers organization cost from the write path to background tasks, matching the append-heavy, rarely-modified log workload. Merges also improve data locality by grouping time-adjacent logs from the same stream, benefiting later queries. However, merges consume CPU, disk bandwidth, and space; if merge speed lags behind ingestion, pressure builds.

Time Partitioning: Don't Search Time-Irrelevant Data

Logs are partitioned by day. A query for the last 10 minutes only scans relevant daily partitions, turning time filtering into a data-location mechanism rather than just a query predicate.

Log Stream Locality: Keep Related Data Together

A log stream groups logs from the same application instance (e.g., service="checkout", instance="node-1"). Storing such logs contiguously improves query locality and compression because repeated field values (service, instance) compress better. The causal chain: better data organization → better locality → fewer irrelevant reads → lower query cost. High-cardinality stream labels must be balanced against storage overhead.

Layer 2: Making Queries Read as Little Data as Possible

Sparse Indexes: Avoid Per-Record Index Overhead

Dense indexes for every record increase write and storage costs. VictoriaLogs uses sparse time indexes—indexing only selected positions—to narrow search ranges while keeping index size manageable.

Bloom Filters: Quickly Prove Data Blocks Can Be Skipped

A Bloom filter per data block probabilistically tests whether a term (e.g., "payment_failed") might exist. "Definitely not present" allows skipping the block; "possibly present" requires reading. False positives are acceptable; false negatives are impossible. For thousands of blocks, skipping even a fraction dramatically reduces total I/O without changing query semantics.

Layer 3: Columnar Storage, Specialized Encoding, and Compression

Why Logs Shouldn't Be Stored as Whole Strings

JSON logs stored row-wise force reading all fields even when querying only one (e.g., counting "level":"error"). Columnar layout stores each field separately, enabling reading only the needed column.

Different Data Types Deserve Different Encodings

Timestamps: Delta encoding (base + differences) exploits continuity. Example: 1000000, 1000001, 1000003 → 1000000, +1, +2, +1.

IP addresses: Store IPv4 as 4 bytes instead of text "192.0.2.10".

Low-cardinality fields (level): Dictionary encoding maps "info"/"error"/"warning" to 0/1/2.

These are conceptual examples, not VictoriaLogs' exact binary formats.

Encoding and Compression Are Partners, Not Alternatives

Specialized encoding removes obvious redundancy and makes data more compressible; general compression (e.g., ZSTD) further reduces size. Smaller data means less disk I/O, better cache utilization, and faster queries under bandwidth constraints. But compression costs CPU; if data is incompressible or frequent decompression is needed, benefits diminish. VictoriaLogs selects per-field encoding strategies and combines them with compression and block organization.

Layer 4: Where Are Go's Performance Boundaries?

GC Isn't the Enemy, But Allocation Behavior Must Be Designed

At 100k logs/sec, even 10 allocations per log creates millions of objects/sec, increasing GC work and memory pressure. The key is not eliminating all allocations but identifying hot paths via profiling. A config parser called 10 times/sec with dozens of allocations is less critical than a field parser called millions of times/sec with few allocations.

Memory Reuse: Reduce Duplicate Work, Don't Blindly Pool

Reusing buffers (e.g., bytes.Buffer) for high-frequency serialization avoids repeated allocation and growth. However, sync.Pool can increase resident memory if large buffers linger, or extend object lifetimes. It should only be used after measurement confirms safety and benefit. Zero-copy byte-to-string conversions require clear ownership to avoid data races.

Measure First, Then Optimize

Go's profiling tools (CPU profile, memory profile, heap allocation, benchmarks) turn "I think this is slow" into "data shows this consumes X resources." A rational optimization loop:

Establish baselines with realistic workloads.

Collect CPU and memory profiles.

Identify the largest cost sources.

First consider architectural/algorithmic elimination of those costs.

Then optimize implementation details.

Re-run benchmarks to verify gains and check for regressions.

From Single Node to Cluster: Performance Isn't Just One Process

VictoriaLogs offers single-node and cluster modes. The cluster splits responsibilities: vlinsert: ingests logs and distributes to storage nodes. vlstorage: persists data and executes query fragments. vlselect: coordinates queries, fans out to storage nodes, merges results.

Horizontal scaling doesn't guarantee linear speedup; data distribution, network overhead, parallelism, and per-node bottlenecks matter. If each node does wasteful reads, scaling just amplifies the waste. Good infrastructure software must both reduce per-task cost (architecture, layout, algorithms) and scale out effectively.

Performance Engineering Lessons for Go Developers

VictoriaLogs' efficiency stems from layered design:

Data architecture minimizes irrelevant data access.

Indexes and Bloom filters skip non-matching blocks.

Columnar storage reads only needed fields.

Specialized encoding + compression cuts storage/transfer cost.

Go runtime allocations, GC, and hot paths are tuned based on measurements.

These layers interact; they collectively determine CPU, memory, and I/O per log from ingestion to query.

A practical optimization sequence:

Re-examine architecture: Can indexes, caching, batching, or better layout reduce work?

Optimize algorithms & data structures: Complexity, locality, encoding, memory access patterns.

Analyze Go runtime costs: Benchmarks and profiles to pinpoint allocations, GC, CPU, copies.

Validate with real workloads: End-to-end throughput, P95/P99 latency, memory peaks, disk I/O, stability.

Continuous verification: Every change needs a hypothesis, reproducible test, and explainable result.

Upgrading Go versions alone won't fix architectural bottlenecks. The biggest performance leap often comes from making code unnecessary, not faster.

References

Go, Logs and the Engineering Behind VictoriaLogs Performance – primary source covering LSM-tree, data locality, Bloom filters, columnar storage, compression, Go memory allocation, and profiling.

https://victoriametrics.com/blog/go-logs-and-the-engineering-behind-victorialogs-performance

Tech Talk #15: Go, Logs and Performance – live walkthrough of storage layout, sharding, and performance practices. https://www.youtube.com/live/2YQVgB3iUyM VictoriaLogs Source Repository – actual storage, query, and merge implementations. https://github.com/VictoriaMetrics/VictoriaLogs VictoriaLogs Key Concepts – log fields, streams, and data model.

https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html

VictoriaLogs Cluster – cluster component roles and data flow. https://docs.victoriametrics.com/victorialogs/cluster Go Official Diagnostics Documentation – profiling tools and methods. https://go.dev/doc/diagnostics VictoriaLogs FAQ – data blocks, columnar layout, encoding, compression, query mechanics.

https://docs.victoriametrics.com/victorialogs/faq/