Building a Custom MySQL Observation Tool with bcc and eBPF

This article, authored by Deng Huan of the iKexing DMP team, demonstrates how to develop a custom MySQL observation tool using the bcc (BPF Compiler Collection) framework and Python. It starts by outlining the need for a bespoke tool when existing bcc utilities do not meet specific monitoring requirements.

Environment Preparation

1. Prepare a Linux machine with matching kernel development packages and Python installed. 2. Deploy a group replication MySQL setup using dbdeployer.

bcc Observation Principle

bcc serves as a front‑end to eBPF, handling data collection through three steps: generating BPF bytecode, loading it into the kernel, and transferring data to user space via perf events or asynchronous methods. The BPF program is written in C and embedded in a Python script.

The following C code is stored in the bpf_text variable:

#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

struct data_t {
    u32 pid;
    u32 tgid;
    u64 ts;
};

BPF_PERF_OUTPUT(events);

int do_apply_data_packet(struct pt_regs *ctx) {
    struct task_struct *t = (struct task_struct *)bpf_get_current_task();
    struct data_t data = {};
    // thread id
    data.pid = t->pid;
    // thread group id (pid in user space)
    data.tgid = t->tgid;
    // timestamp in nanoseconds since boot
    data.ts = bpf_ktime_get_ns();

    events.perf_submit(ctx, &data, sizeof(data));
    return 0;
}

This program captures the current task's PID, TGID, and a timestamp, then submits the data to a perf buffer named events .

Attaching the Probe

The script locates the MySQL group replication plugin's apply_data_packet function and attaches the BPF program using attach_uprobe :

# get mysql function name and trace it.
path = "/root/sandboxes/mysql_base/8.0.18/lib/plugin/group_replication.so"
regex = "\\w+apply_data_packet\\w+"
symbols = BPF.get_user_functions_and_addresses(path, regex)
if len(symbols) == 0:
    print("Can't find function 'apply_data_packet' in %s" % (path))
    exit(1)
(mysql_func_name, addr) = symbols[0]
b = BPF(text=bpf_text)
b.attach_uprobe(name=path, sym=mysql_func_name, fn_name="do_apply_data_packet")

If kernel‑space functions need tracing, attach_kprobe would be used instead.

Outputting Results

# output trace result.
print("Tracing MySQL server mgr apply_data_packet function")
print("%-14s %-6s %-6s" % ("SINCE_UP_TIME(s)", "PID", "THREAD"))

def print_event(cpu, data, size):
    event = b["events"].event(data)
    print("%-14s %-6s %-6s" % (event.ts/1000000000, event.tgid, event.pid))

b["events"].open_perf_buffer(print_event)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit()

The script prints a header, then continuously reads events from the perf buffer and displays the elapsed time since boot, the process ID, and the thread ID.

Result Demonstration

Running the tool and generating MySQL traffic produces output similar to:

root@ubuntu:/tmp# python mgr_apply_data_packet.py
Tracing MySQL server mgr apply_data_packet function
SINCE_UP_TIME(s) PID    THREAD
2165924        26387  27060
2165924        25810  27043
2165924        26962  27080

References

1. http://www.brendangregg.com/ebpf.html 2. https://github.com/iovisor/bcc/blob/master/docs/tutorial_bcc_python_developer.md 3. https://github.com/iovisor/bcc/blob/7e3f0c08c7c28757711c0a173b5bd7d9a31cf7ee/tools/dbslower.py 4. https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md