Mastering Log Monitoring with Wonder: Types, Strategies, and Go Implementation

Background

Log monitoring means watching application logs, extracting the data you care about, and setting alert policies so that when thresholds are crossed an alarm is generated. The Wonder monitoring system follows the HULK private‑cloud platform’s service‑oriented philosophy and is built on an Open‑Falcon‑based architecture.

Log Monitoring Types

Rolling Count

Scenario: Used to monitor changes in log entry counts, such as QPS spikes.

Method: Count the number of log lines that appear within a monitoring interval. Choose "Rolling Count" as the monitoring type, specify the log path, and set the collection frequency.

String Matching

Scenario: Match keywords in logs to trigger alerts; applicable to most application logs.

Method: Choose between exact match and regular‑expression match. Configure whether repeated occurrences count as one, define the match content, select the log path, and set the collection frequency.

Numeric Matching

Scenario: Perform range matching on numeric fields in logs and compute sum, min, max, avg (e.g., alert when nginx request_time avg > 10 s).

Method: In a monitoring interval, match all numbers that satisfy the rule, then calculate max, min, average, and total. Configure start/end strings; the system inserts a numeric regex (‑?\d+\.?\d*) to capture the value.

Log Monitoring Principle

1. Log Reading – Uses the Go tail library (github.com/hpcloud/tail) with a small modification. When an agent receives a log‑monitoring configuration, it initializes the monitor if it does not exist and starts reading the log file.

<code>t, err := tail.TailFile(filename, tail.Config{Follow: true, ReOpen: true, Debug: debug, Poll: true, Location:&amp;tail.SeekInfo{Whence: os.SEEK_END}})</code>

The agent launches a goroutine that continuously reads lines and passes each line to FormateMsg for processing.

<code>for line := range t.Lines {
    _, err := t.Tell()
    if err != nil {
        if g.Config().Debug {
            log.Println(err)
        }
        return
    }
    FormateMsg(w, line.Text)
}</code>

2. Message Formatting – FormateMsg parses the log according to its type and configuration, then pushes the processed value into a queue.

<code>w.Queue.PushFront(r)</code>

3. Periodic Processing – Another goroutine periodically processes the queued data and pushes the final result to the transfer layer.

<code>go func() {
    for {
        select {
        case <-this.Ticker.C:
            RunTask(this.Log)
        case <-this.Quit:
            this.Ticker.Stop()
            return
        }
    }
}()</code>

4. Transfer Layer – Data is tagged with the log path to support multiple logs under a single monitoring configuration.

Conclusion

The current log‑monitoring features meet basic business needs, and future work will add capabilities such as date‑based log file matching. As log volume grows, CPU consumption increases, so large‑file handling may switch to an SDK‑based push model similar to ELK for centralized collection and processing.