How Heka + ELK Supercharges Mobile Game Log Management

Outline

序

源起

什么是ELK？

ELK能做什么？

手游接入的价值

手游如何做接入

日志关键字报警

未来的方向

序

Current market demands for operations, maintenance, and R&D capabilities are rising; rapid response to evolving business processes is essential, yet existing data analysis often falls short. Big data, real‑time analytics, and machine learning tools can significantly improve response speed and automated insight.

This is the core of AIOps platforms: big data. Today we focus on data collection, storage, analysis engines, and visualization, and the value they bring to the business.

源起

Logs are simple text files that developers and operators interact with daily, providing system health monitoring, rapid issue location, faster customer response, higher software availability, fault root‑cause tracking, and continuous optimization.

Historically, log analysis required manual SSH access and tools like AWK/GREP, making the process cumbersome, especially for large log volumes.

With the rise of distributed systems, larger clusters, and higher user expectations, centralized real‑time log analysis platforms have become urgent. Open‑source ELK has become widely adopted alongside commercial solutions like Splunk.

ELK’s advantages are low integration cost, fast speed, and quick results, making it a key weapon in our AIOps platform for games.

什么是ELK？

ELK is the abbreviation of three open‑source components: Elasticsearch, Logstash, and Kibana.

Elasticsearch : a distributed search engine with features such as zero‑configuration, auto‑discovery, automatic index sharding, and easy cluster management.

Logstash : an open‑source log collection tool for gathering and processing logs.

Kibana : a web UI that lets users create dashboards, perform searches and aggregations with just a few clicks.

Since version 5.0, Elastic renamed the ELK Stack to Elastic Stack, adding the Beats suite.

ELK能做什么？

ELK enables rapid statistics of server log errors, custom client log reporting, and real‑time monitoring through Kibana dashboards.

手游接入的价值

Integrating ELK into mobile games provides centralized log collection, fast error detection, and actionable insights that improve game stability and player experience.

手游如何做接入？

Mobile client logs are sent via HTTP + URL‑encoded POST. On the server side, configure an HTTP Post Input to receive these logs.

Step 1: Enable Heka HttpListenInput

Step 2: Use Heka sandbox plugin to run Lua for custom log parsing

In cz_decoder.lua you can extract fields, handle missing IPs by using HTTP RemoteAddr, and optionally capture the user‑agent.

最快的接入

Heka can ingest standard Nginx logs in about one second.

Step 1: Check Nginx log_format

Step 2: Edit /etc/hekad.toml to match Nginx format and set the Type

Step 3: Search logs in Kibana

还有哪些有意思的优化？

1. Disable Elasticsearch auto‑refresh to reduce memory usage and schedule a custom refresh via cron.

2. Extract real error file names from exception_info in Lua for deduplication.

3. Add basic security to Kibana via Nginx configuration or X‑Pack (paid after trial).

日志关键字报警

Using Yelp’s open‑source ElastAlert, you can set up keyword‑based alerts for Elasticsearch 5.0+ indices.

When conditions are met, ElastAlert sends an email notification.

未来的方向

We are enhancing the architecture by adding queue clusters for stable ingestion, improving overall performance, integrating internal authentication, encrypting client reports, building alert configuration UI, and linking log data with business data for richer insights.

Ultimately, we aim to maximize the value of ELK in the AIOps era, supporting ever‑changing business needs and delivering greater efficiency for the company.