Why OTA Updates Trigger Device Reboots and How to Fix OOM Crashes

What Is OTA?

OTA (Over‑The‑Air) is a method for upgrading a device’s system without physical loss, allowing new features to be deployed remotely. Updates can be downloaded directly over the network or via an OTA package placed on an SD card or USB drive.

OTA Download Methods

SMS method

PUSH method

Network customization

The example in this article uses the network‑customized method.

Phenomenon Description

During OTA upgrade testing, the product was upgraded 162 times, resulting in 19 crashes/reboots. The following screenshots illustrate the crash statistics.

Crash Analysis

1. Why Does OOM Cause Reboot?

When a process requests a physical page, the fast path is tried first; if it fails, the slow path is used. If both fail, the OOM‑killer is triggered. Normally the OOM‑killer terminates the process consuming the most pages, but with panic_on_oom=1 the kernel panics and reboots.

Even when many free pages exist, allocation can fail due to severe memory fragmentation or inability to borrow other migration‑type pages. Disabling panic_on_oom avoids the panic, but the parameter is often enabled by default, leading to the observed reboots.

When panic_on_oom is disabled, the OOM‑killer kills the process with the largest physical page usage (e.g., the Bluetooth process), which does not affect the upgrade workflow.

2. Why Are Many Free Pages Still Leading to OOM?

Although logs show about 126 MiB of free physical pages, allocation still fails because the gfp_mask (0x101cc0) does not set MIGRATE_MOVABLE, preventing the use of ALLOC_CMA pages. Consequently, cma_pages are unavailable.

When CMA pages are disallowed, the usable pages equal free - free_cma. Although free_cma reports over 120 MiB (about half of total memory), the remaining non‑migratable pages are only a few megabytes, leading to allocation failures.

Solution

In addition to disabling panic_on_oom, investigate why free_cma can be allocated so heavily without a maximum limit. System logs show the CMA maximum limit is 128 MiB. Reducing CMA to 64 MiB frees memory for non‑migratable page allocation.

Set the CMA size via U‑Boot parameter: cma=64M, which limits CMA to 64 MiB.