LAW: Log Asynchronous Writer – A Go Library for Asynchronous Logging Across Multiple Loggers

The author, a veteran with 17 years in the IT industry, describes the motivation behind creating LAW (Log Asynchronous Writer), a Go library that adds standardized asynchronous output to various logging libraries without requiring modifications to the libraries themselves.

Project Goals

Standardize asynchronous output for multiple Golang log libraries.

Require minimal code changes and low learning cost.

Provide a user‑friendly interface that is simple, efficient, and high‑performance.

Ensure future compatibility for quick integration of new log libraries.

Project Overview

LAW (https://github.com/shengyanli1982/law) is designed to let any Go logging library support asynchronous writes simply by importing the library. It already adapts zap , logrus , klog , and zerolog , and more adapters are planned.

The library exposes only two interfaces:

Write([]byte) : works like io.Writer , allowing direct use wherever an io.Writer is accepted.

Stop() : stops the asynchronous routine and releases resources.

Architecture Design

To achieve true asynchrony, LAW decouples log production from log output using a channel. It reduces system‑call overhead by buffering writes with bufio . A dedicated goroutine consumes the channel and writes to a buffered writer, while another goroutine monitors idle time and flushes the buffer when necessary.

The core components are:

channel : separates log production from consumption.

bufio writer: minimizes syscall calls.

Performance Comparison

A simple benchmark compares synchronous logging (direct os.Stdout ) with asynchronous logging via LAW . Results show the async version consistently achieves 83K‑86K requests per second versus 76K‑80K for the sync version, roughly a 10% throughput increase and lower latency variance.

Usage Example

The following demo shows how to integrate LAW with zap :

package main

import (
    "net/http"
    "os"

    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"

    x "github.com/shengyanli1982/law"
)

func main() {
    aw := x.NewWriteAsyncer(os.Stdout, nil)
    defer aw.Stop()

    encoderCfg := zapcore.EncoderConfig{
        MessageKey: "msg",
        LevelKey:   "level",
        NameKey:    "logger",
        EncodeLevel: zapcore.LowercaseLevelEncoder,
        EncodeTime:  zapcore.ISO8601TimeEncoder,
        EncodeDuration: zapcore.StringDurationEncoder,
    }

    zapSyncWriter := zapcore.AddSync(aw)
    zapCore := zapcore.NewCore(zapcore.NewJSONEncoder(encoderCfg), zapSyncWriter, zapcore.DebugLevel)
    zapLogger := zap.New(zapCore)

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        zapLogger.Info("hello")
    })
    _ = http.ListenAndServe(":8080", nil)
}

Running the demo prints JSON‑formatted log entries asynchronously, as shown by the sample output.

Code Walkthrough

The library’s double‑buffer implementation uses a channel to queue log entries, a buffer pool to reuse memory, and a goroutine ( poller ) that writes buffered data to the underlying writer. A second goroutine ( bufferIoWriterRefresh ) flushes the buffer when it becomes idle for more than the default timeout.

// poller reads log entries from the queue and writes them to the underlying writer
func (wa *WriteAsyncer) poller() {
    defer wa.wg.Done()
    for eb := range wa.queue {
        bytes := eb.Buffer().Bytes()
        now := time.Now().UnixMilli()
        wa.config.cb.OnPopQueue(bytes, now-eb.UpdateAt())
        wa.bufferIoLock.Lock()
        _, err := wa.buffWriter(bytes)
        wa.bufferIoLock.Unlock()
        if err != nil {
            log.Printf("data write error, error: %s, message: %s", err.Error(), util.BytesToString(bytes))
        }
        wa.idleAt.Store(now)
        wa.bufferPool.Put(eb)
    }
}

// bufferIoWriterRefresh flushes the buffer when idle time exceeds the threshold
func (wa *WriteAsyncer) bufferIoWriterRefresh() {
    heartbeat := time.NewTicker(time.Second)
    defer func() { heartbeat.Stop(); wa.wg.Done() }()
    for {
        select {
        case <-wa.stopCtx.Done():
            return
        case <-heartbeat.C:
            wa.bufferIoLock.Lock()
            if wa.bufferIoWriter.Buffered() > 0 && time.Now().UnixMilli()-wa.idleAt.Load() > defaultIdleTimeout.Milliseconds() {
                if err := wa.bufferIoWriter.Flush(); err != nil {
                    log.Printf("buffer io writer flush error, error: %s", err.Error())
                }
            }
            wa.bufferIoLock.Unlock()
        }
    }
}

Conclusion

By providing a lightweight, easy‑to‑use asynchronous writer that can be plugged into any Go logging library, LAW achieves significant performance gains while keeping the integration effort minimal. The author invites the community to adopt, test, and contribute to further improvements.