Mastering Byteman Helpers: Custom Rules, Lifecycle Hooks, and Compilation Strategies

User-Defined Rules and Helpers

Byteman allows rules to be extended, overridden, or replaced by specifying a custom helper class. In the example below, the FailureTester helper class determines whether to throw a WrongStateException based on its doWrongState(CoordinatorEngine) method.

# Helper example
RULE help yourself
CLASS com.arjuna.wst11.messaging.engines.CoordinatorEngine
METHOD commit
HELPER com.arjuna.wst11.messaging.engines.FailureTester
AT EXIT
IF doWrongState($0)
DO throw new WrongStateException()
ENDRULE

Helper Class Requirements

The helper class must not be declared final so that Byteman can subclass it.

It cannot be abstract because Byteman needs to instantiate it.

It must provide a public constructor. By default, Byteman uses the no‑argument constructor (), but if a constructor (org.jboss.byteman.agent.rule.Rule) exists, it will be preferred.

Configuring Helpers for Multiple Rules

By adding a HELPER line in a rule file, you can set the default helper for subsequent rules.

HELPER HelperSub
RULE helping hand
... 
RULE I can't help myself
... 
HELPER YellowSub
RULE help, I need somebody
CLASS ...
METHOD ...
...

The first two rules use HelperSub, while the third uses YellowSub.

Helper Lifecycle Methods

Helper classes can implement lifecycle methods that are invoked automatically when rules are loaded or unloaded.

public static void activated()
public static void deactivated()
public static void installed(Rule rule)
public static void installed(String ruleName)
public static void uninstalled(Rule rule)
public static void uninstalled(String ruleName)

Lifecycle Event Hooks

activated

: called when the rule set transitions from empty to non‑empty, suitable for one‑time initialization. deactivated: called when the rule set becomes empty, suitable for resource cleanup. installed: called when a rule is loaded, useful for registering listeners or initializing rule‑specific data structures. uninstalled: called when a rule is removed, allowing cleanup of rule‑related resources.

Lifecycle Method Call Chain

Byteman ensures that lifecycle methods are invoked following the helper class inheritance hierarchy:

Installation events : start from the rule's helper class and call activated and installed up the inheritance chain.

Uninstallation events : start from the helper superclass and call uninstalled and deactivated down the chain.

Target and Trigger Classes

The CLASS or INTERFACE clause defines the target class or interface for a rule. Matching can occur in several ways:

Exact class name matches multiple classes : e.g., CLASS Foo may apply to org.my.Foo and org.acme.Foo.

Subclass matching : e.g., CLASS ^Foo matches org.my.FooBar and org.my.FooBaz.

Interface matching implementations : e.g., INTERFACE Foo applies to any class implementing that interface.

Two type‑checking modes control how the match is performed:

AS TARGET (lexical scope): checks the target type. Example:

AS TARGET
IF $0.append($2)

If the target type List does not define append, the check fails.

AS TRIGGER (dynamic scope, default): checks the runtime type of the injection point, allowing subclass‑specific methods.

Specifying AS TARGET or AS TRIGGER in a rule file gives precise control over type safety and rule applicability.

Rule Compilation Options

By default, rules are interpreted, but they can be compiled to bytecode for better performance. Compilation can be controlled globally or per rule.

Global default : set the system property org.jboss.byteman.compile.to.bytecode or use COMPILE / NOCOMPILE directives in the rule file.

Per‑rule compilation :

RULE compile example
CLASS com.arjuna.wst11.messaging.engines.CoordinatorEngine
METHOD prepare
COMPILE
AT ENTRY
...
ENDRULE

Rule‑group mode control :

COMPILE
RULE example 1
...
ENDRULE
NOCOMPILE
RULE example 2
...
ENDRULE

Using COMPILE forces bytecode compilation, while NOCOMPILE forces interpretation, allowing a balance between performance and loading overhead.