Handling Large Arrays in PHP: File and Database Processing Strategies

When performing data statistics, you often encounter large arrays that can cause memory overflow. This article discusses how to handle large arrays in PHP, covering both large file processing and large‑dataset database reads.

Large File Processing

The core idea for handling huge files is to read them line by line, which allows processing without loading the entire file into memory.

Large file processing core idea: read line by line.

A recommended article provides a concrete implementation (see the linked guide).

Database Large‑Data Processing

When reading massive data from a database, common problems include:

Data volume too large to fetch.

Large arrays that cannot be processed.

For the first issue, optimize the database configuration or the SQL query, e.g., narrow the query range and process data in batches.

Core idea for DB large‑data handling: destroy variables promptly after use.

Because looping over large arrays consumes a lot of memory, releasing variables as soon as they are no longer needed helps avoid memory overflow.

How to destroy variables promptly

$sql = "your sql";
  $rs = $DB->query($sql);
  $data = array();
  foreach($rs as $v){
    // your code
  }
  unset($rs); // destroy variable

In practice, unsetting the loop variable inside foreach does not free memory until the loop finishes. The article demonstrates a project where reading ~700,000 rows caused memory exhaustion.

Project Practice

Background: a performance‑scoring module needs to read millions of rows and aggregate metrics per employee.

Problem: when the data set is large, the script crashes before the unset at the end of the loop can run.

Solution: try unsetting each element inside the loop and monitor memory usage.

set_time_limit(0);
ini_set('memory_limit', '1024M');
echo "\r\nbefore‑sql:" . memory_get_usage();
$sql = "my sql";
$rs = M()->query($sql);
$data = array();
echo "\r\nbefore‑data:" . memory_get_usage();
foreach ($rs as $k => $value) {
    // calculation logic
    ...
    $data[$user_id][$key]['reg'] += $value['reg_num'];
    $data[$user_id][$key]['fee_money'] += $value['ad_fee_usd'];
    $data[$user_id][$key]['pay_money'] += $value['usd_money'];
    echo "\r\nmid‑before‑arr:" . memory_get_usage();
    print_r($rs[1]);
    unset($rs[$k]);
    echo "\r\narr‑count:" . count($rs);
    echo "\r\nmin‑final‑arr:" . memory_get_usage();
}
echo "\r\nfinal‑data:" . memory_get_usage();

Results showed that unsetting elements inside foreach did not significantly reduce memory usage because PHP's garbage collector only frees the memory after the whole array is destroyed.

Switching to a for loop with an external count() allowed memory to be released gradually:

set_time_limit(0);
ini_set('memory_limit', '1024M');
// ... same setup as above
$num = count($rs);
for ($k = 0; $k < $num; $k++) {
    $value = $rs[$k];
    // calculation logic
    ...
    $data[$user_id][$key]['reg'] += $value['reg_num'];
    // ... other aggregations
    echo "\r\nmid‑before‑arr:" . memory_get_usage();
    print_r($rs[1]);
    unset($rs[$k]);
    echo "\r\narr‑count:" . count($rs);
    echo "\r\nmin‑final‑arr:" . memory_get_usage();
}
echo "\r\nfinal‑data:" . memory_get_usage();

The output confirmed that memory decreased as the array elements were unset, demonstrating PHP's garbage collection behavior.

Additional Tips

Use file_get_contents for small files; for large files, read line by line with fgets to avoid memory spikes.

foreach is generally faster than for , but when you need to modify the collection during iteration, a for loop with a pre‑computed count can be more efficient.

Perform large‑array processing in background jobs or scheduled tasks to improve user experience.

Conclusion

Both large‑file handling and large‑dataset DB reads trade time for space. Choose the approach that fits your business constraints, and apply proper variable destruction and loop strategies to manage memory effectively.