Building a High‑Performance, Secure Mobile Logging System with Mars XLog

Dev Club is a community for mobile developers where experts share technical topics. In this session, Tencent WXG Android senior engineer Yan Guoyue presented the high‑performance logging module xlog of WeChat's Mars framework.

1. Mars Overview

Mars is a C++‑based, platform‑agnostic foundation component used by WeChat on Android, iOS, macOS, Windows, and WP. It provides common libraries such as comm (socket, thread, message queue, coroutine), xlog (logging), sdt (network diagnostics), and stn (signal distribution).

Mars is used by over 800 million monthly active WeChat users, offering both long‑connection and short‑connection monitoring.

2. Why a New Logging Solution?

Traditional Android logging writes encrypted text to files for every line, causing heavy GC and I/O overhead, leading to UI jank. Release builds often disable logging, reducing debugging efficiency.

The main performance bottleneck is frequent file writes, which involve copying data to kernel dirty pages and later flushing to disk under conditions such as timed writeback, memory pressure, or low memory.

SSD write‑amplification further worsens latency, as whole blocks must be erased and rewritten.

3. xlog‑V1.0 Design

Logs are first buffered in memory, then compressed and encrypted before being written to a file. This reduces I/O but still incurs CPU spikes during batch compression and can lose logs if the process crashes before flushing.

4. xlog‑V2.0 Design

4.1 mmap

Using mmap maps a file into memory, allowing direct memory writes without extra copies and giving better control over write‑back timing (memory pressure, process exit, msync / munmap, or periodic sync on some OSes).

4.2 Compression

The module uses phrase‑based (LZ77) compression followed by Huffman coding, achieving about 86 % reduction. Four schemes were tested; the best (stream‑oriented) kept a 83.7 % compression ratio while limiting the impact of corrupted data to the current compression unit.

4.3 Final xlog Scheme

Each log line is compressed, encrypted, and written into an mmap‑backed buffer; when the buffer reaches a threshold it is flushed to disk.

This approach balances smoothness, completeness, fault tolerance, and security, with an acceptable 83.7 % compression rate and negligible CPU impact.

5. Key Requirements for Mobile Logging

Never log user‑private data in plaintext.

Logging must not cause UI jank.

Logs must survive process kills and crashes.

Partial file corruption should not invalidate the entire log.

6. Mars Open‑Source Plan

Mars is expected to be open‑sourced by the end of the year, with the same source used internally at WeChat.

Interactive Q&A

Q1: How does the crash‑capture module work?

It is not part of the open‑source Mars; for C++ crashes, refer to Android backtrace and libunwind. Java crash handling is outside the scope of this talk.

Q2: Does the log require any server‑side support?

All logging occurs on the client; there is no mandatory server interaction.

Q3: Are there JNI performance concerns when calling C++ from Java?

Early Android versions had JNI overhead, but recent optimizations have made the cost negligible.

Q4: What is the purpose of the log header’s magic number?

It distinguishes log versions and helps locate the start of each compressed, encrypted block.

Q5: Are logs ever uploaded to a server?

Typically logs stay on the device; users can manually upload them if needed, e.g., via a specific command.

Q6: Can custom data be stored in the log?

Only string representations of data should be logged; the log format itself is not extensible.

Q7: When is encryption applied?

In V2.0 each line is compressed then encrypted before being written to mmap; there is no size‑threshold batching.

Q8: Is log upload part of the open‑source code?

No; uploading involves retry logic, chunking, and server‑side storage, which are outside the core xlog module.

Q9: How does Mars differ from Bugly?

Bugly focuses on crash reporting, while Mars provides logging, signaling, network diagnostics, and other cross‑platform C++ utilities.

Q10: How is the in‑memory log buffer size determined?

It is fixed at 150 KB, based on empirical compression tests, and does not adapt per device.

Q11: Are compression and encryption available as separate APIs?

The open‑source release does not expose them separately; developers can implement their own if needed.

Q12: Why are Android‑specific optimizations discussed?

Android’s aggressive process killing makes log loss more likely, so special handling is required.

Q13: Can logs be stored in a database instead of files?

Storing logs in a database is discouraged due to performance overhead; the module aims to stay lightweight (~120 KB SO).

Q14: What does “compression unit” mean in the fourth scheme?

It defines the range of data over which compression state is reset, limiting the impact of corruption to that unit.