Using Chronicle Queue for High-Performance Persistent Queues in Java

Queues are fundamental components in software design patterns, but handling millions of messages per second and allowing multiple consumers to read all messages poses challenges such as memory pressure and JVM garbage collection.

The author previously built timestamp‑based log replay engines that stored logs in memory, which worked for up to ten million entries (~1 GB) with acceptable GC behavior, but scaling further requires disk‑based storage and more efficient parsing.

Chronicle Queue, an open‑source library offering terabyte‑scale file I/O with low latency, is introduced as a solution. The article demonstrates how to create a massive persistent queue while keeping latency predictable.

First, a FunLog class extending SelfDescribingMarshallable is defined to hold URL, host, and timestamp fields:

private static class FunLog extends SelfDescribingMarshallable {
    String url
    String host
    int time
    FunLog() {}
    FunLog(String url, String host, int time) {
        this.url = url
        this.host = host
        this.time = time
    }
}

An initial attempt using ConcurrentLinkedQueue fails because each element creates a wrapper node, doubles object count, exhausts heap memory, cannot be accessed from other JVMs, and loses data when the JVM exits.

Chronicle Queue is then used for both writing and reading billions of log entries. Writing example:

static void main(String[] args) {
    String basePath = getLongFile("chronicle");
    ChronicleQueue queue = ChronicleQueue.singleBuilder(basePath).build();
    def appender = queue.acquireAppender();
    int total = 1_0000_0000;
    def start = Time.getTimeStamp();
    total.times {
        def log = new FunLog(Time.getDate(), index.getAndIncrement() + EMPTY, getMark());
        appender.writeDocument(log);
    }
    def end = Time.getTimeStamp();
    output(total / (end - start) * 1000);
    output(queue.lastIndex() - queue.firstIndex());
}

Reading example:

static void main(String[] args) {
    String basePath = getLongFile("chronicle");
    ChronicleQueue queue = ChronicleQueue.singleBuilder(basePath).build();
    def tailer = queue.createTailer();
    def log = new FunLog();
    int total = 1_0000_0000;
    def start = Time.getTimeStamp();
    total.times {
        tailer.readDocument(log);
    }
    def end = Time.getTimeStamp();
    output(total / (end - start) * 1000);
    output(queue.lastIndex() - queue.firstIndex());
}

Performance tests show that writing 100 million entries consumes about 4.5 GB of disk, achieves roughly 1.7 million QPS, and reading sustains around 1.6 million QPS, while keeping JVM memory and GC impact low by reusing a single FunLog object.

The article also references related resources on Java serialization optimization, generic object pooling, and other performance topics.