How to Eliminate High‑Latency Decoding in Moonlight Android for Smooth Game Streaming

The author, a handheld‑gaming enthusiast, explored the open‑source Moonlight project and optimized its Android video decoding module for smoother game streaming.

Game Streaming Overview

Game consoles like PS4 can stream to handhelds or mobile devices, and PC games can be streamed similarly using apps such as Steam Link, Nvidia GameStream, or AMD's solutions, which encode video frames directly from the GPU.

Moonlight

Moonlight is an open‑source implementation of Nvidia's GameStream protocol, built on the C‑based moonlight-common-c library. It receives streamed video, decodes it, and presents it with low latency, similar to cloud‑gaming services.

Android Video Decoding

Android offers two decoding APIs: the Java MediaCodec and the native C AMediaCodec . Decoding can be performed via active buffer requests or passive callbacks; the former is faster and recommended.

High‑Latency Decoding Issue

High‑latency decoding occurs when decoded frames are returned hundreds of milliseconds after they are ready, often due to internal buffer congestion. While tolerable for media playback, it is unacceptable for real‑time game streaming, prompting Moonlight to drop one frame to avoid the delay.

How to Avoid High‑Latency Decoding

Do not use the Java API for decoding, as vendor‑specific optimizations can make latency control difficult.

Immediate Release

Release decoded buffers as soon as they are displayed, avoiding blocking calls such as AMediaCodec_releaseOutputBuffer (introduced in API 21), which may wait for vertical sync and trigger high latency.

<code>media_status_t AMediaCodec_releaseOutputBuffer(AMediaCodec*, size_t idx, bool render) __INTRODUCED_IN(21);</code>

Use the non‑blocking alternative:

<code>media_status_t AMediaCodec_releaseOutputBufferAtTime(AMediaCodec *mData, size_t idx, int64_t timestampNs) __INTRODUCED_IN(21);</code>

Frame Ordering

At 60 fps, each frame should be displayed after roughly 16.67 ms. Because callback timing varies (0–20 ms), delaying frame presentation by one frame provides a buffer to absorb decoding and transmission delays.

Configuration Changes

Some decoders require vendor‑specific low‑latency extensions. Example for Qualcomm devices:

<code>// Set the Qualcomm vendor low latency extension if the Android R option is unavailable
if (MediaCodecHelper_decoderSupportsQcomVendorLowLatency(decoderName)) {
    AMediaFormat_setInt32(videoFormat, "vendor.qti-ext-dec-low-latency.enable", 1);
}

// HiSilicon (Huawei) low‑latency settings for Kirin chips
if (MediaCodecHelper_decoderSupportsHisiVendorLowLatency(decoderName)) {
    AMediaFormat_setInt32(videoFormat, "vendor.hisi-ext-low-latency-video-dec.video-scene-for-low-latency-req", 1);
    AMediaFormat_setInt32(videoFormat, "vendor.hisi-ext-low-latency-video-dec.video-scene-for-low-latency-rdy", -1);
}

if (maxOperatingRate) {
    AMediaFormat_setInt32(videoFormat, "operating-rate", 32767); // Short.MAX_VALUE
}
</code>

Adjusting the H.264 SPS profile to baseline (profile_idc = 66) can also reduce B‑frame buffering.

<code>sps.profile_idc = 66;</code>

Improvements to Moonlight

The author’s custom Moonlight build limits frame delay to three frames and rewrites the decoder in C, enabling stable 120 fps streaming without frame drops, outperforming the official version.

Download the official app from Google Play or the modified version from the GitHub releases page.

Related Code

Official repository: https://github.com/moonlight-stream/moonlight-android

Author’s fork: https://github.com/rexq57/moonlight-android

Java decoding example (image omitted for brevity).

C decoding example (image omitted for brevity).