Master Java Microbenchmarking with JMH: A Hands‑On Guide

Introduction

Simple main programs used to compare algorithm performance suffer from high variance, interference, and lack of warm‑up. JMH (Java Microbenchmark Harness), bundled with JDK 9+, provides a reliable framework for method‑level micro‑benchmarking with nanosecond precision.

What Is JMH?

JMH is a suite designed for micro‑benchmarks. It runs at the method level, offers nanosecond‑level accuracy, and is authored by the developers of the JVM JIT compiler, ensuring deep awareness of JIT effects.

Typical Use Cases

Measure stable execution time of a method and its scaling with input size.

Compare throughput of different implementations under identical conditions.

Determine the percentage of requests completed within a given time frame.

Getting Started

Dependency

For JDK 9+ JMH is included. For earlier JDKs add the following Maven dependencies (latest version 1.27 at the time of writing):

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.27</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.27</version>
</dependency>

Sample Benchmark

The benchmark below compares StringBuffer and StringBuilder for three input lengths (10, 50, 100):

// Benchmark configuration
@BenchmarkMode(Mode.AverageTime)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 3, time = 4)
@Threads(1)
@Fork(1)
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class JmhTest {

    @Param({"10", "50", "100"})
    private int length;

    @Benchmark
    public void testStringBufferAdd(Blackhole bh) {
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < length; i++) {
            sb.append(i);
        }
        bh.consume(sb.toString());
    }

    @Benchmark
    public void testStringBuilderAdd(Blackhole bh) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < length; i++) {
            sb.append(i);
        }
        bh.consume(sb.toString());
    }

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(JmhTest.class.getSimpleName())
                .result("result.json")
                .resultFormat(ResultFormatType.JSON)
                .build();
        new Runner(opt).run();
    }
}

Running the Benchmark

Executing the main prints JMH version, JVM details, warm‑up configuration, measurement configuration, thread count, and the selected benchmark mode. Warm‑up iterations are not counted in the final results; they allow the JIT compiler to optimise the code before measurement.

Typical truncated output (nanoseconds per operation) looks like:

Benchmark                     (length)  Mode  Cnt   Score   Error  Units
JmhTest.testStringBufferAdd      10   avgt    3  92.599 ± 105.019  ns/op
JmhTest.testStringBuilderAdd    100   avgt    3 903.055 ± 294.557  ns/op

Common Annotations

@BenchmarkMode : Selects measurement mode(s) – Throughput, AverageTime, SampleTime, SingleShotTime, or All.

@Warmup : Configures warm‑up iterations, duration, time unit, and optional batch size.

@Measurement : Mirrors @Warmup but for the actual measurement phase.

@State : Defines the lifecycle of benchmark class instances (e.g., Scope.Thread, Scope.Benchmark, Scope.Group).

@OutputTimeUnit : Sets the time unit for reported results.

@Threads : Number of threads that execute the benchmark concurrently.

@Fork : Number of separate JVM processes to launch for isolation.

@Param : Supplies a set of values for a field, enabling parameterised benchmarks.

@Benchmark : Marks the method to be measured.

@Setup / @TearDown : Run before/after each benchmark iteration for resource preparation and cleanup.

Important Pitfalls

Dead‑Code Elimination

The compiler may remove code that appears unused, skewing results. Use Blackhole.consume or return a non‑void value to force evaluation.

Constant Folding

If inputs are compile‑time constants, the JIT may pre‑compute results. Read inputs from a @State field or return the computed value to avoid this optimisation.

Loop Unwinding

JIT may unroll loops, affecting measurement. Prefer to let JMH handle loop overhead via @OperationsPerInvocation or use @BenchmarkMode(SingleShotTime) with an appropriate batchSize.

Result Visualization

The benchmark writes a JSON file ( result.json) containing all console output. This file can be visualised with external tools such as JMH Visual Chart (http://deepoove.com/jmh-visual-chart/) or JMH Visualizer (https://jmh.morethan.io/).

Packaging as an Executable JAR

For large‑scale or CI testing, shade JMH into a runnable JAR using Maven's shade plugin:

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.1</version>
        <executions>
            <execution>
                <phase>package</phase>
                <goals><goal>shade</goal></goals>
                <configuration>
                    <finalName>jmh-demo</finalName>
                    <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>org.openjdk.jmh.Main</mainClass>
                        </transformer>
                    </transformers>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

Build and run the JAR:

mvn clean package
java -jar target/jmh-demo.jar JmhTest

Conclusion

The article outlines the essential concepts, annotations, best practices, and tooling required to perform accurate micro‑benchmarks with JMH. By configuring warm‑up, measurement, forks, and using Blackhole or return values to avoid dead‑code elimination and constant folding, developers can obtain reliable performance data and integrate benchmarking into CI pipelines.