Build a Real‑Time Log Collector with Java and tail – Why -F Beats -f

Real‑Time Log Collection with tail

The tail command can continuously output new lines appended to a file. By launching tail -f (or -F) from Java with ProcessBuilder, a program can read each line in real time and forward it to any destination, such as a message queue. The example below simply prints each line.

import java.io.BufferedReader;
import java.io.InputStreamReader;

public class TailReader {
    public static void main(String[] args) throws Exception {
        ProcessBuilder pb = new ProcessBuilder("tail", "-f", "/tmp/tail0");
        pb.redirectErrorStream(true);
        Process p = pb.start();
        try (BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()))) {
            String line;
            while ((line = in.readLine()) != null) {
                setLogToKafka(line);
            }
        }
    }
    static void setLogToKafka(String line) {
        System.out.println(line);
    }
}

If the tail process is terminated, the Java collector stops receiving data, which is a risk that must be handled in production.

Difference Between tail -f and tail -F

Log rotation typically renames the current log file and creates a new one. tail -f follows the original file descriptor, so after rotation it continues to read the renamed file and misses new entries. tail -F follows the filename, automatically retrying when the file is recreated, and therefore works correctly with rotation.

Example Logback configuration that rolls a file nightly:

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <prudent>true</prudent>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>logFile.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>
      <totalSizeCap>3GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
    </encoder>
  </appender>
  <root level="DEBUG">
    <appender-ref ref="FILE" />
  </root>
</configuration>

To simulate a rotation manually:

mv run.log run.2020-11-02.log
touch run.log

Step‑by‑Step Test

Create the file to monitor: touch /tmp/tail0 Run the Java program shown above.

Generate a continuous stream of timestamps: watch -n 1 'date >> /tmp/tail0' Simulate log rotation:

mv /tmp/tail0 /tmp/tail.bak
touch /tmp/tail0

After the rotation, a process started with tail -f stops receiving new data because it is still attached to /tmp/tail.bak. The following commands illustrate the state:

ps -ef | grep tail
# example output: 12345 6789 0 1:51PM ?? 0:00.01 tail -f /tmp/tail0

lsof -p 12345 | awk '{print $4 "\t" $9}'
# shows a file descriptor pointing to /tmp/tail.bak

Writing to the old file restores output, confirming that -f follows the descriptor. Replacing -f with -F makes tail follow the filename, so the Java program continues to receive lines after rotation.

Additional Considerations

If a file is deleted while still open by a process, the data remains on disk until the process exits. Use lsof | grep deleted to locate such handles.

To truncate a log file without deleting it (avoiding dangling descriptors), use cat /dev/null > logfile or the shell shortcut : > logfile instead of rm.

Conclusion

Understanding the subtle behavior of tail -f versus tail -F prevents silent log‑collection failures during file rotation. The lightweight Java wrapper demonstrates how to build a custom real‑time log collector using standard Unix tools.