Designing and Implementing an Effective Log System for Internet Startups

For an internet startup, the business systems generate massive amounts of logs daily—system logs, transaction logs, exception logs, access logs, audit logs, etc. Collecting these logs enables both offline and online analysis to support business operations and operational insights.

Beyond analysis, developers rely on logs to troubleshoot issues, as logs often provide the only evidence of what happened at a specific moment. Logs also help monitor runtime state during development.

Three Parts of a Log System

Log Generation : Client‑side and server‑side logs capture events for troubleshooting, runtime visibility, and analysis.

Log Collection : Distributed logs must be aggregated from various machines and services.

Log Analysis : Collected logs are processed for analysis, alerting, or direct developer inspection.

Log Levels

According to RFC 5424, there are eight levels (Emergency, Alert, Critical, Error, Warning, Notice, Informational, Debug). In practice, many teams simplify to four levels:

Error : Critical failures that must be addressed immediately.

Warning : Situations that allow continued operation but require attention.

Info : Important business events, such as successful user logins.

Debug : Verbose diagnostic information, typically used only in development.

Different environments use different default levels: Production (Info or higher), Testing (Debug when needed), Development (Debug).

Essential Log Fields

Version

Timestamp (or NULL if unavailable)

App Name (identifies device or application)

Message Type (semantic classification)

Message Content (actual log message)

Important Additional Fields

Result

Consumed Time

Source (e.g., IP, caller)

Location/Path (file and line number for precise tracing)

Logging Principles

Coloring System : Assign a globally unique log ID (logid) to each request, propagate it through all layers, and include it in every log entry to enable end‑to‑end traceability.

Structured Data : Record IP, timestamp, business context, machine, service, class/function, and optionally line number, ensuring completeness via a logging library.

Human‑Readable Messages : Write logs in clear language that conveys who, what, where, and why, avoiding ambiguous phrases.

Log Collection Methods

When logs are emitted in a standard format, they must be gathered for downstream use. Simple setups write logs to local text files; more complex setups use tools like ELK.

Example ELK architecture:

LogStash Agent on each machine uploads logs to a queue.

Redis acts as a buffering queue.

Logstash cluster parses logs into JSON and forwards them to Elasticsearch.

Elasticsearch stores logs for real‑time search and aggregation.

Kibana provides visual exploration and reporting.

Another approach is a dedicated log service consisting of:

Writer : SDK used by applications to emit logs.

Relay (Agent) : Forwards logs and can persist locally if needed.

Collector : Receives logs from agents and routes them.

Store : Persists logs, often on Hadoop for offline analysis, and may also feed real‑time systems like Storm.

Even a simple UDP socket service can serve as a minimal log collector.

Post‑Collection Uses

Collected logs enable monitoring, real‑time search, and issue localization. With ELK, logs can be:

Routed to a monitoring service that triggers alerts and creates Jira tickets.

Indexed in Elasticsearch for ad‑hoc queries and Kibana dashboards.

Fed to Kafka for downstream real‑time analytics (e.g., Storm).

Used with colored log IDs to quickly locate related events across services.

Recommendation for Startups

For early‑stage startups, a straightforward setup—local file logging combined with an ELK stack—covers most needs and scales for future growth. Implement a standard logging library, consider a coloring system when resources allow, and evolve the architecture as the business expands.

Reference: http://diyitui.com/content-1432833903.30823746.html