Uncovering Hidden JVM Bugs with Classfuzz and Classming Bytecode Mutations

Background

The Java Virtual Machine (JVM) must produce identical results for the same class across different implementations, yet testing JVMs is difficult because their input is compiled bytecode. Traditional TCK tests require Oracle’s proprietary suite, which is unavailable to most researchers.

Goal: Expose JVM Defects

The work aims to answer two questions: (1) How to discover JVM bugs or security flaws, and (2) How to generate effective test inputs—numerous, executable bytecode variants—to exercise JVM implementations.

Detect differences between expected and actual JVM behavior by running the same class on multiple JVMs and comparing results.

Generate a large set of diverse bytecode files that are syntactically valid and semantically varied.

Classfuzz: Syntax‑Level Bytecode Mutation

Classfuzz applies simple syntactic transformations to a seed class, such as changing public to private, renaming methods, or altering file names. These mutations produce many unusual classes that can be fed to JVMs to test their robustness.

Initial experiments showed that Classfuzz could trigger format errors in HotSpot and uncover validation bugs in OpenJ9. For example, a class with an interface flag but missing the required abstract flag caused HotSpot to reject the class while OpenJ9 accepted it, revealing a discrepancy.

Technical Points of Classfuzz

129 mutation operators were designed, 123 targeting syntax (e.g., visibility changes, method deletions) and 6 targeting semantics.

Semantic mutations use the Soot framework to convert classes to Jimple, then reorder or modify statements (e.g., swapping two Jimple instructions).

Operator selection is guided by a Markov‑Chain Monte Carlo algorithm that favors operators observed to be more effective.

Representative test classes are chosen via traditional equivalence‑class partitioning and coverage metrics (line and branch coverage) on a target JVM.

Differential testing runs each mutated class on multiple JVMs, using majority voting to infer which JVM exhibits the fault.

Classming: Domain‑Aware Semantic Mutation

Classming goes beyond syntax by applying domain‑aware changes based on Java bytecode characteristics. Starting from a seed class, the tool inserts or modifies control‑flow constructs (e.g., goto, return, throw) and data‑flow elements, aiming to produce bytecode that remains executable but behaves differently.

Examples include inserting loops around monitor instructions, swapping initialization order, or altering object types (e.g., changing a Map to a String) to provoke verification or execution differences between HotSpot and OpenJ9.

Findings from Differential Testing

HotSpot and OpenJ9 differ in handling of uninitialized objects used in entermonitor / exitmonitor sequences, leading to distinct exceptions (IMSE vs. NullPointerException).

Verification rules such as the “of no consequence” clause for <clinit> methods are interpreted differently, exposing spec ambiguities.

Version‑specific bugs were discovered, e.g., a class that passes verification in one HotSpot version but fails in another due to structural‑lock checks.

Overall Impact and Future Work

The combined Classfuzz and Classming framework provides a systematic approach to generate valid yet diverse Java bytecode, enabling differential testing that uncovers JVM specification ambiguities, implementation bugs, and potential security issues. Future directions include applying the generated variants to stress memory management and performance subsystems of JVMs.

Collaborators on this research include Prof. Su Zhendong (ETH Zurich), Prof. Zhao Jianjun (Kyushu University), Dr. Su Ting (Nanyang Technological University), and Dr. Sun Chengnian (Google).