How to Process a 10 GB CSV File with Only 512 MB RAM?

An interview asks how to preprocess a 10 GB CSV file when only 512 MB of RAM and a single‑core CPU are available, expecting a solution beyond the obvious line‑by‑line read.

Initial ideas such as reading each line directly, using multithreading, or loading the data into a database are dismissed as either too trivial or too resource‑intensive for the constraints.

The author first suggests dividing the file into blocks, recording the byte offset of each block, processing a chunk (e.g., 10,000 rows), writing the transformed data to a new file, and then jumping to the next offset, thus avoiding loading the entire file into memory.

After the interview, the author discovers the standard answer: stream the CSV line by line, optionally batch a few thousand rows, keep memory usage minimal, and implement a checkpoint mechanism that records the last processed line number or byte offset so the job can resume after a failure.

Implementation in Java can use BufferedReader or libraries like OpenCSV/Super CSV. Example code for simple line processing:

BufferedReader reader = new BufferedReader(new FileReader("large.csv"));
String line;
while ((line = reader.readLine()) != null) {
    processLine(line); // conversion, validation, etc.
    writeToOutput(line); // add to new file
}

If parsing is expensive, batch processing of 1,000–5,000 lines is possible, ensuring the batch size stays well within the available memory:

List<String> batch = new ArrayList<>(5000);
while ((line = reader.readLine()) != null) {
    batch.add(line);
    if (batch.size() == 5000) {
        processBatch(batch);
        batch.clear(); // release memory
    }
}

For checkpointing, a small file can store the current byte offset, e.g.,

offset = 1048576  # current processed bytes

When the program restarts, RandomAccessFile can seek to the saved offset and continue processing:

RandomAccessFile raf = new RandomAccessFile("large.csv", "r");
raf.seek(lastKnownOffset); // continue from here

An optional producer‑consumer optimization creates two threads: one reads lines into a small queue (producer) and the other consumes the queue for processing (consumer). On a single‑core machine this may be overkill, but it illustrates how to overlap I/O and computation.

If the CSV resides on a distributed file system and a cluster of low‑spec nodes is available, Kafka + Flink (or Spark Streaming) become appropriate, providing data sharding, parallel processing, and exactly‑once semantics; however, they are unnecessary for a single‑node task.

Key take‑aways: clarify the problem scope before proposing heavyweight frameworks, ask about the execution environment, and prioritize fault‑tolerant, low‑memory streaming with checkpointing over raw throughput.