Tagged articles
Kernel Tracing
23 articles · Page 1 of 1
This article provides a comprehensive guide to Linux kernel tracing using Ftrace and kprobe, covering static and dynamic instrumentation, practical command workflows, kernel module examples, real-world debugging case studies for network latency and memory errors, and critical performance overhead considerations.
DebuggingFtraceKernel Module
0 likes · 24 min read

HUATUO, Didi's open‑source cloud‑native observability project, leverages BPF‑based low‑overhead kernel tracing, unified metric and event frameworks, automatic flame‑graph generation, and seamless integration with Prometheus, Grafana and Elasticsearch to provide panoramic, zero‑intrusive monitoring and continuous performance profiling for complex production environments.
BPFKernel Tracingcloud-native
0 likes · 11 min read

Network latency spikes often leave developers guessing whether the culprit lies in user‑space, the kernel stack, or the physical link; this article introduces TencentOS’s NBS (Net Blackboard System), a low‑overhead, zero‑disruption solution that pinpoints delay sources, supports continuous deployment, and outperforms traditional tools like tcpdump and bpftrace.
Kernel TracingNBSOperations
0 likes · 14 min read

The article introduces Linux’s built‑in performance analysis tool perf, explains its event‑driven sampling, tracing and profiling capabilities, shows how to install it on various distributions, demonstrates common commands with real code examples, and discusses practical scenarios for locating and optimizing kernel and application performance issues.
Kernel TracingSystem Optimizationbenchmarking
0 likes · 36 min read

Last Branch Record (LBR) is a CPU‑level feature that records branch jumps; the Linux kernel’s bpf_get_branch_snapshot helper (since 5.16) enables eBPF programs to capture LBR data, and the bpflbr tool demonstrates how to trace kernel functions and bpf program execution, disassemble code, and output call stacks.
Kernel TracingLast Branch RecordLinux
0 likes · 9 min read

The article explains how to add runtime‑configurable filtering of kernel function arguments in eBPF programs by parsing a C‑style expression, validating its AST, converting it to BPF instructions using BTF metadata, and injecting the generated code into the probe, with a complete example for skb filtering.
BPFBTFGo
0 likes · 15 min read

This article provides a comprehensive overview of Linux kernel tracing mechanisms—including probes, tracepoints, kprobes, uprobes, and ftrace—explaining their principles, implementation details, practical tools, and real‑world case studies for performance optimization, fault diagnosis, and security monitoring.
DebuggingFtraceKernel Tracing
0 likes · 36 min read

This article explains how to use eBPF and perf events to trace NFS read/write system calls, capture kernel and user stack traces, resolve them to source locations via DWARF, and generate flame‑graph data that is pushed to Pyroscope for performance analysis.
Flame GraphKernel TracingNFS
0 likes · 16 min read

This article demonstrates how to write an eBPF program that attaches to Linux soft‑interrupt entry and exit points, records timestamps in eBPF maps, computes handling duration, updates counters and histograms, and exposes the data to user space for performance analysis.
Kernel TracingLinuxPerformance Monitoring
0 likes · 5 min read

The article explains why backtracing with eBPF fentry on arm64 is harder than on x86, details the stack layout differences, shows how recent commits changed register saving, and provides a practical detection routine to locate the frame pointer and retrieve the tracee's instruction pointer.
BPFKernel Tracingarm64
0 likes · 5 min read

This article introduces rtla timerlat, a Linux scheduling‑latency analysis tool that breaks latency into detailed phases, explains how to cross‑compile its libtracefs and libtraceevent dependencies, compile the tool, enable the kernel timerlat tracer, configure common parameters, and run a typical command with sample output.
Cross-compilationKernel TracingLinux
0 likes · 11 min read

This article demonstrates how to use eBPF together with BTF to trace BPF map update and delete functions, shows concrete command‑line output, explains the code that identifies target kernel functions, and details the data‑dumping logic for debugging map contents.
BPF mapBTFDebugging
0 likes · 8 min read

This article explains how to monitor Go goroutine state transitions without modifying source code by attaching eBPF uprobes to the runtime.casgstatus function, defining a custom data struct, using a ring buffer to deliver events, and processing them in a Go user‑space program.
GoKernel TracingeBPF
0 likes · 16 min read

This article explains how to use eBPF fentry and fexit probes to monitor the Linux unlink system call, compares them with kprobes, provides complete C and Go examples, shows compilation with eunomia‑bpf, and discusses the low‑overhead advantages of fentry for kernel tracing.
C++GoKernel Tracing
0 likes · 9 min read

This article explains how to use Linux eBPF kprobe (and kretprobe) to dynamically instrument the unlink system call, covering the underlying concepts, required kernel headers, full eBPF source code, compilation with both eunomia‑bpf and cilium/ebpf, and a detailed comparison with tracepoint probes.
Kernel TracingLinuxeBPF
0 likes · 17 min read

This article walks through creating a first eBPF program using C and Go on Ubuntu 22.04, covering required dependencies, kernel‑space vs user‑space concepts, event selection, BPF map definition, and a tracepoint function that measures per‑process CPU time.
BPF mapsC++Go
0 likes · 11 min read

This article guides you through installing BCC, creating simple and complex eBPF programs with C and Python, explains eBPF architecture, source‑code organization, and demonstrates real‑world tracing examples for kernel diagnostics and performance monitoring.
BCCKernel TracingPerformance Monitoring
0 likes · 12 min read

The article describes an eBPF‑based, language‑agnostic traffic recording framework that hooks low‑level socket operations and thread identifiers to capture complete request‑response flows across Java, PHP, and Go services without modifying application code, dramatically lowering implementation and maintenance costs for cloud‑native traffic replay.
GoKernel TracingSocket
0 likes · 15 min read

eBPF, a safe, high‑performance Linux kernel extension evolving from the 1993 Berkeley Packet Filter to modern dynamic tracing, underpins Didi’s HuaTuo platform, which consolidates bytecode management, fast data processing, stability self‑healing, and container insight to solve traffic replay, topology, security, and root‑cause analysis challenges across cloud‑native services, with plans to broaden business use and community collaboration.
HuatuoKernel TracingRoot Cause Analysis
0 likes · 12 min read

The article details how Linux’s function‑graph ftrace tracer works on ARM64, explaining required kernel configs, how -pg inserts _mcount calls, the runtime patching of ftrace_graph_caller, register usage for argument passing, and return handling, and why shadow‑call‑stack must be disabled to enable precise call‑graph and timing analysis.
FtraceKernel TracingLinux
0 likes · 10 min read

This article walks through the high‑level components of eBPF programs—backend, loader, frontend, and data structures—showing how the original sock_example.c is split into separate files, how LLVM compiles restricted C to ELF, and how projects like BCC, BPFtrace, and IOVisor automate development, deployment, and cloud‑native observability while highlighting their trade‑offs for embedded environments.
BCCKernel TracingLinux
0 likes · 15 min read

This article explains the principles behind Android Systrace, details its implementation using ftrace, and shows how to add custom trace categories and parsers via Catapult to visualize kernel‑level information in Chrome's tracing UI.
AndroidCatapultCustom Trace
0 likes · 8 min read

TcpRT is a real‑time instrumentation and diagnostic system for Alibaba Cloud RDS that non‑intrusively collects TCP trace data, aggregates billions of records per day, applies statistical and Cauchy‑based anomaly detection, and pinpoints root causes across hosts, proxies, and network devices at massive scale.
Kernel TracingSIGMODanomaly detection
0 likes · 27 min read
