Design and Implementation of Qunar's Non‑Intrusive Online Debug System

Introduction

Debugging has long been a pain point for online applications, often leading developers to repeatedly add log statements and redeploy. To address this, Qunar initially built a cloud‑based Debug tool that required new components and restarts, which eventually fell out of favor.

Background

After reflecting on the shortcomings of the first version—dependency on new components, configuration changes, restart requirements, and reliance on internal tracing—the team released a new online Debug that works transparently with both legacy and new systems, requiring no additional dependencies or restarts.

Overview

The online Debug is a feature of Qunar's production issue diagnosis platform Bistoury. The UI receives user requests and forwards them to a Proxy, which adjusts commands (e.g., fetching the target process ID) and sends them to an Agent running on the user’s machine. The Agent communicates with the user application, loads the Debug module via Java Instrumentation, and returns results back through the same chain.

Principle

Java bytecode retains source line numbers, allowing the system to locate the exact bytecode location for a breakpoint. Using the ASM framework, the tool injects snapshot‑capture logic at the breakpoint, enabling on‑the‑fly debugging.

userSystem.preDo();

userSystem.do();

userSystem.afterDo();

After instrumentation, the code checks hasBreakpoint() and, if true, calls captureSnapshot() before proceeding with the original method.

Runtime Instrumentation

Java's Instrumentation API (available since JDK 1.6) allows an external Agent JAR to be loaded into a running JVM. The Agent attaches to the target JVM, loads the Instrumentation JAR, obtains an Instrumentation instance, and starts a Netty server to handle Debug requests.

BistouryClassLoader

To avoid version conflicts between the attach JAR’s dependencies (e.g., Guava) and the user application, a dedicated BistouryClassLoader loads the attach JAR. It overrides loadClass to prefer its own classes for non‑JDK types, while the lightweight Instrumentation JAR is loaded by the BootstrapClassLoader.

Serialization Issues

When capturing a snapshot, objects must be serialized to JSON without invoking getters (which may have side effects) and while handling circular references. Standard JSON libraries either call getters (FastJSON) or require annotations (Jackson, Gson). The solution modifies Jackson’s source within the attach JAR to safely serialize arbitrary objects.

MagicClassLoader

A second custom loader, MagicClassLoader, ensures that classes from Bistoury‑magic‑classes.jar have priority over others, enabling quick fixes for buggy third‑party libraries without waiting for upstream releases.

Conclusion

The article presents the design and implementation of Qunar’s non‑intrusive online Debug system, inviting colleagues to try it and provide feedback.