Design and Implementation of a Real-Time Log Collection and Query System for Distributed Deployment

When diagnosing problems in distributed systems, engineers often have to log into multiple machines to view logs, which is time‑consuming and error‑prone, especially for long call chains.

For the Zz release system, the existing tracing tool only supported Java services, while the deployment pipeline also involved Go‑based agents, and the sheer number of servers (thousands) made per‑machine log access cumbersome due to permission and login overhead.

The deployment process consists of seven steps—backup, download, unpack, environment init, nginx config removal, service start, and nginx re‑enable—each requiring commands sent from the release service to the agent, which then executes scripts on target machines and reports results back.

To address these issues, the team evaluated commercial and open‑source log solutions and built a near‑real‑time log system centered on Elasticsearch (the "E" of the ELK stack). Java services push logs via the Elasticsearch client, while Go agents report via HTTP endpoints, and Kibana provides a searchable UI for the entire deployment workflow.

In practice, the index mapping is defined (e.g., GET zzdeploylog/_mapping) and logs can be queried by a unique task ID to reconstruct the full deployment sequence, showing command parameters and outcomes for each step, thus eliminating the need to manually log into each host.

The new system has significantly improved troubleshooting efficiency; future plans include ingesting Docker container logs, enhancing Elasticsearch cluster stability, and extending the solution to other business services.