Master Java Performance Testing with JMH: From Setup to Advanced Benchmarks

1. Introduction

JMH (Java Microbenchmark Harness) is a Java tool for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java or other JVM languages.

Why use JMH?

Accuracy : JMH is dedicated to micro‑benchmarking, pre‑warming the code and allowing configuration of process and thread counts to reflect real‑world performance.

Targeted : It operates at the method level, letting developers measure execution time of specific methods and understand the impact of different inputs.

Functionality : Supports various measurement modes such as Throughput, AverageTime, SampleTime, and SingleShotTime to suit different testing scenarios.

Ease of Use : Provides a rich API and configuration options, integrates with Maven and Gradle, and can be added to existing projects.

Note: The recommended way to run JMH benchmarks is to create a separate Maven project that depends on the application JAR, ensuring proper initialization and reliable results.

2. Practical Examples

2.1 Build with Maven

Run the following command to generate a JMH project:

<code>mvn archetype:generate \
  -DinteractiveMode=false \
  -DarchetypeGroupId=org.openjdk.jmh \
  -DarchetypeArtifactId=jmh-java-benchmark-archetype \
  -DgroupId=com.pack \
  -DartifactId=jhm-demo \
  -Dversion=1.0</code>

After Maven downloads the dependencies, the project is created successfully.

Update the JMH version in pom.xml to the latest (e.g., 1.37):

<code>&lt;jmh.version&gt;1.37&lt;/jmh.version&gt;
&lt;dependency&gt;
  &lt;groupId&gt;org.openjdk.jmh&lt;/groupId&gt;
  &lt;artifactId&gt;jmh-core&lt;/artifactId&gt;
  &lt;version&gt;${jmh.version}&lt;/version&gt;
&lt;/dependency&gt;</code>

2.2 Build and Run the Jar

Compile the project:

<code>mvn clean verify</code>

The generated JAR is located in the target directory.

2.3 Run the Benchmark

Execute the self‑contained JAR:

<code>java -jar target/benchmarks.jar</code>

2.4 Direct Main‑Method Execution

Alternatively, run benchmarks from a main method:

<code>@Benchmark
public void testMethod() {
  // your test code
}

public static void main(String[] args) throws Exception {
  Options options = new OptionsBuilder()
      .include(MyBenchmark.class.getSimpleName())
      .forks(1)
      .build();
  new Runner(options).run();
}</code>

This approach is convenient but slightly less accurate than the jar method.

2.5 Core Annotations

<code>// Warm up for 10 iterations, 1 second each
@Warmup(iterations = 10, time = 1)
// Fork 1 JVM with specific heap settings
@Fork(value = 1, jvmArgsAppend = {"-Xms512m", "-Xmx512m"})
// Measure average time per operation
@BenchmarkMode(Mode.AverageTime)
// Output results in nanoseconds
@OutputTimeUnit(TimeUnit.NANOSECONDS)
// Share state across all benchmark threads
@State(Scope.Benchmark)
public class MyBenchmark {
  // TODO
}
</code>

These annotations control warm‑up, forking, measurement mode, time unit, and state scope.

2.6 Dealing with Dead Code

Methods that return a value are slower because the JIT cannot eliminate the dead code:

<code>// Returns a result (uses variable i)
@Benchmark
public int testMethodReturnResult() {
  int i = 0;
  i++;
  return i;
}

// No return (dead code can be optimized away)
@Benchmark
public void testMethodNoReturn() {
  int i = 0;
  i++;
}
</code>

2.7 Consuming Values with Blackhole

When benchmarking multiple variables, use Blackhole to prevent dead‑code elimination:

<code>@Benchmark
public void testMethodMultiVariable(Blackhole hole) {
  int i = 0;
  i++;
  int j = 0;
  j++;
  hole.consume(i);
  hole.consume(j);
}
</code>

2.8 @Setup Annotation

@Setup runs before each benchmark method:

<code>@Setup
public void init() {
  System.out.println("Preparing before test...");
}
</code>

2.9 Date‑Format Benchmark

<code>@Benchmark
public void testSimpleDateFormat(Blackhole hole) {
  SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
  String ret = sdf.format(new Date());
  hole.consume(ret);
}

@Benchmark
public void testDateTimeFormatter(Blackhole hole) {
  String ret = LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"));
  hole.consume(ret);
}
</code>

2.10 Map Traversal Benchmark

<code>private List<UserDTO> datas = new ArrayList<>();

@Setup
public void init() {
  for (int i = 0; i < 100_000; i++) {
    datas.add(new UserDTO(i + 0L, "Name - " + i));
  }
}

@Benchmark
public void testStream(Blackhole hole) {
  List<User> ret = datas.stream()
      .map(dto -> new User(dto.id, dto.name))
      .collect(Collectors.toList());
  hole.consume(ret);
}

@Benchmark
public void testParallelStream(Blackhole hole) {
  List<User> ret = datas.parallelStream()
      .map(dto -> new User(dto.id, dto.name))
      .collect(Collectors.toList());
  hole.consume(ret);
}
</code>

2.11 SpringBoot Integration Benchmark

<code>private PersonService ps;
private ConfigurableApplicationContext context;

@Setup
public void init() {
  context = SpringApplication.run(SpringbootDbApplication.class);
  ps = context.getBean(PersonService.class);
}

@Benchmark
public void testSave() {
  Person person = new Person();
  int num = new Random().nextInt(100);
  person.setAge(num);
  person.setName("Name - " + num);
  this.ps.save(new Person(77, "Hey"));
}

@Benchmark
public void testQuery(Blackhole hole) {
  Person p = this.ps.findById(30);
  hole.consume(p);
}

@TearDown
public void down() {
  this.context.close();
}
</code>

In summary, JMH provides a reliable way to measure Java code performance, offering accurate warm‑up, flexible measurement modes, and seamless integration with Maven, Gradle, and SpringBoot projects.