System Log Architecture: Designing Logs That Answer 'Who Did What When'

Chapter Overview

The core challenge after an incident is not a lack of logs but an abundance of logs that cannot answer "who did what to which business at what time." This chapter classifies logs into nine types—login, logout, menu, function, process, business, interface, program, and tracking—and presents a centralized logging architecture covering collection, cleaning, storage, and analysis. The key principle: different logs answer different questions, so purpose and data structure must be defined before choosing collection and storage methods.

Key Knowledge Breakdown

6.1–6.3 Login and Logout Logs

Login logs capture who, when, where, via which device and method, and which system—supporting anomaly detection, activity analysis, and user self-service. Logout logs are not strictly one-to-one with logins; closing the client, session expiry, and forced kickout carry different semantics, so they should not be forced into a single login record by updating it.

Security-related logs must record result, reason, and context, but never raw passwords, full tokens, or other secrets. When displaying login history to clients, IP, location, and device information should be appropriately masked.

6.4 Operation Traces

Menu logs reflect entry and dwell time; function logs capture specific actions; process logs link multi-page steps; business logs record data state changes. Together they form a user behavior timeline from coarse to fine.

Business operation logs should adopt an event mindset: record business primary key, event type, before/after summary, operator, and timestamp. For high-value data, logging full payloads can cause volume and privacy issues; instead, save necessary fields, diffs, or controlled snapshots.

6.5 Interface Logs

Interface logs should be recorded uniformly via filters, interceptors, or aspects—capturing request entry, response exit, latency, result code, and correlation identifiers—to minimize business code intrusion. Distinguish business failures from system exceptions, and impose limits on request body size, binary content, and sensitive fields.

A single call must carry a TraceID, a local SpanID, and upstream correlation data to reconstruct the transaction chain across frontend, gateway, services, and asynchronous messages.

6.6–6.7 Program Logs and Storage

Log levels control information density; production should retain key business nodes, warnings, and errors, avoiding long-term debug output. Logging must not share transactions with the main business; use buffering, async queues, or agent collection, but critical audit logs must define explicit loss policies.

Storage planning must consider capacity, retention, indexing, archiving, and deletion. Structured fields enable filtering and statistics; raw messages aid diagnosis; combining both is more effective than plain text alone.

6.8 Centralized Collection Architecture

A centralized log platform can be abstracted into collection, transport/buffering, cleaning, storage, and analysis. Lightweight agents like Filebeat suit collection; message queues provide peak shaving and isolation; processors like Logstash handle parsing and cleaning; engines like Elasticsearch provide search; UIs like Kibana handle visualization. Smaller scale can simplify the pipeline; add buffering and processing layers as volume and reliability demands grow.

Architecture Diagrams

Diagram 1 illustrates the log platform data pipeline. Diagram 2 emphasizes that asynchronous messages must also propagate correlation identifiers; otherwise the full chain breaks at the queue.

Personal Insights

"Logs are the query model for future problems" reframes log design from output formatting to rehearsing future troubleshooting: after an incident, will we search by user, task, artifact, or environment? Those correlations must be established today.

In an R&D platform, a deployment issue may trace from runtime environment back to artifact, pipeline instance, code commit, and requirement. Without stable correlation IDs, a centralized log platform only offers faster full-text search, not automatic business relationship reconstruction.

The book's log classification need not be copied verbatim. User operations, platform runtime, and build tool outputs face different audiences, retention periods, and sensitivity levels; classification should be re-derived from purpose.

A more practical question than "what to log" is "what must never be logged." Build commands, environment variables, and tool outputs can leak credentials. The notion that more complete logs always aid debugging holds only within privacy and security boundaries.

Practical Implementation Considerations

Log Standards

Define unified fields for timestamp, environment, service, instance, TraceID, SpanID, user, and business primary key.

Establish an audit event dictionary for critical events such as login, permission changes, fund movements, and data modifications.

Prohibit or mask logging of passwords, tokens, keys, ID numbers, bank card numbers, and similar sensitive fields.

Unify business error codes and system exception categories to ensure consistent alerting and statistics.

Set tiered retention periods, hot/cold tiering, compression, archiving, and deletion mechanisms.

Runtime Validation

Select a real transaction spanning multiple services and message queues; verify that a user-visible symptom can be traced to every hop, business state change, and final result. Then simulate failures in the collection agent, buffering layer, or storage to confirm that system performance, log loss scope, and recovery process meet agreed SLAs.