How to Process a 16 GB Log File in Seconds with Go

Modern computer systems generate massive logs every day. Storing such immutable debugging data in a database is impractical, so most companies keep logs as files on local disks. This article shows how to extract logs from a 16 GB .txt / .log file using Go.

First, the file is opened with the standard os.Open call and proper error handling:

f, err := os.Open(fileName)
if err != nil {
    fmt.Println("cannot able to read the file", err)
    return
}
// UPDATE: close after checking error
defer file.Close() // Do not forget to close the file

Three reading strategies are considered: line‑by‑line (low memory, slower), loading the whole file (fast but impossible for 16 GB), and chunked reading with bufio.NewReader. The third option balances memory usage and speed.

r := bufio.NewReader(f)
for {
    buf := make([]byte, 4*1024) // the chunk size
    n, err := r.Read(buf) // loading chunk into buffer
    buf = buf[:n]
    if n == 0 {
        if err != nil {
            fmt.Println(err)
            break
        }
        if err == io.EOF {
            break
        }
        return err
    }
    nextUntillNewline, err := r.ReadBytes('
') // read entire line
    if err != io.EOF {
        buf = append(buf, nextUntillNewline...)
    }
    wg.Add(1)
    go func() {
        // process each chunk concurrently
        ProcessChunk(buf, &linesPool, &stringPool, &slicePool, start, end)
        wg.Done()
    }()
}
wg.Wait()

To reduce garbage‑collector pressure, sync.Pool is used for reusable buffers, strings, and slices. Each chunk is processed in a separate goroutine, dramatically improving throughput.

linesPool := sync.Pool{New: func() interface{} { return make([]byte, 500*1024) }}
stringPool := sync.Pool{New: func() interface{} { return "" }}
slicePool := sync.Pool{New: func() interface{} { return make([]string, 100) }}
var wg sync.WaitGroup
for {
    buf := linesPool.Get().([]byte)
    n, err := r.Read(buf)
    buf = buf[:n]
    if n == 0 {
        if err != nil { fmt.Println(err); break }
        if err == io.EOF { break }
        return err
    }
    nextUntillNewline, err := r.ReadBytes('
')
    if err != io.EOF { buf = append(buf, nextUntillNewline...) }
    wg.Add(1)
    go func(chunk []byte) {
        ProcessChunk(chunk, &linesPool, &stringPool, &slicePool, start, end)
        wg.Done()
    }(buf)
}
wg.Wait()

The ProcessChunk function splits each chunk into lines, parses the ISO‑8601 timestamp, and prints the line only if the timestamp falls between the user‑provided start and end times.

func ProcessChunk(chunk []byte, linesPool *sync.Pool, stringPool *sync.Pool, start time.Time, end time.Time) {
    logs := string(chunk)
    linesPool.Put(chunk)
    logsSlice := strings.Split(logs, "
")
    for _, text := range logsSlice {
        if len(text) == 0 { continue }
        parts := strings.SplitN(text, ",", 2)
        tsStr := parts[0]
        ts, err := time.Parse("2006-01-02T15:04:05.0000Z", tsStr)
        if err != nil { fmt.Printf("
 Could not parse time: %s for log: %v", tsStr, text); return }
        if ts.After(start) && ts.Before(end) {
            fmt.Println(text)
        }
    }
}

The full program parses command‑line arguments ( -f "From Time" -t "To Time" -i "Log file path"), determines the file size, reads the last line to check whether the requested time range overlaps the file, and then calls Process to handle the data. Execution time is printed at the end.

func main() {
    s := time.Now()
    args := os.Args[1:]
    if len(args) != 6 {
        fmt.Println("Please give proper command line arguments")
        return
    }
    startTimeArg := args[1]
    finishTimeArg := args[3]
    fileName := args[5]
    file, err := os.Open(fileName)
    if err != nil { fmt.Println("cannot able to read the file", err); return }
    defer file.Close()
    queryStartTime, _ := time.Parse("2006-01-02T15:04:05.0000Z", startTimeArg)
    queryFinishTime, _ := time.Parse("2006-01-02T15:04:05.0000Z", finishTimeArg)
    // ... (logic to locate last line and its timestamp) ...
    if lastLogCreationTime.After(queryStartTime) && lastLogCreationTime.Before(queryFinishTime) {
        Process(file, queryStartTime, queryFinishTime)
    }
    fmt.Println("
Time taken - ", time.Since(s))
}

func Process(f *os.File, start time.Time, end time.Time) error {
    // pools and buffered reading as shown earlier
    // launch goroutines that call ProcessChunk
    return nil
}

Benchmarking the solution on a 16 GB log file shows that the extraction completes in roughly 25 seconds, demonstrating that Go’s concurrency primitives and memory‑reuse techniques make processing massive log files both fast and memory‑efficient.