How to Monitor Log Files in Real-Time with Python: 3 Simple Methods

In environments where service stability is critical, operations teams need to detect problematic requests instantly rather than waiting for failures. Monitoring Nginx variables such as $request_time and $upstream_response_time in real time helps identify the slowest calls, allowing developers to optimise code promptly. This article presents three practical Python approaches for real‑time log file monitoring.

Method 1: Invoke tail -f via subprocess.Popen

The simplest solution leverages the familiar Unix tail -f command. By spawning a subprocess, the script reads each new line that matches a pattern (e.g., "timeout") and prints it.

logfile='access.log'

command='tail -f '+logfile+'|grep "timeout"'

popen=subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

while True:
    line=popen.stdout.readline().strip()
    print line

Method 2: Direct File Positioning with tell() and seek()

This approach works purely with Python’s file API. It repeatedly records the current file offset, reads a line, and if no new line is available, sleeps briefly and seeks back to the saved offset, effectively waiting for new data.

import time

file = open('access.log')

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        print line,

Method 3: Generator with yield for Continuous Streaming

A more Pythonic solution defines a generator that yields each new line as it appears. The generator seeks to the end of the file, then loops, sleeping briefly when no data is available, and yields lines when they arrive.

import time

def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

if __name__ == '__main__':
    logfile = open("access-log","r")
    loglines = follow(logfile)
    for line in loglines:
        print line,

The seek() function moves the file pointer by a specified offset. Its signature is file.seek(off, whence=0), where whence determines the reference point: 0 for the file start, 1 for the current position, and 2 for the end of the file.

These three techniques can be adapted to any log‑analysis scenario; developers should choose the method that best fits their performance requirements and code‑base conventions.