HarmonyOS VR Player: Building an Industrial-Grade NDK Demuxing Pipeline

Introduction: The Art of Demultiplexing

In VR scenarios with 4K/8K resolution, demuxing performance directly impacts startup latency and seek responsiveness. This chapter dives into HarmonyOS NEXT's Media Kit to build an industrial-grade NDK demuxing pipeline for the QVRPlayer.

Data Source Construction: OH_AVSource Integration

1. Universal Path Support: Local to Network

QVRPlayer uses OH_AVSource_CreateWithURI which supports both local sandbox paths and HTTP/HTTPS network streams.

// Core logic: create Source instance
mSource = OH_AVSource_CreateWithURI(const_cast<char*>(url.c_str()));
if (!mSource) {
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_DOMAIN, LOG_TAG, "Failed to create source from URI: %{public}s", url.c_str());
    return -1;
}

2. Global Metadata Parsing

After source creation, extract file overview such as duration and bitrate:

auto sourceFormat = OH_AVSource_GetSourceFormat(mSource);
int64_t duration = 0;
OH_AVFormat_GetLongValue(sourceFormat, OH_MD_KEY_DURATION, &duration);
OH_LOG_Print(LOG_APP, LOG_INFO, LOG_DOMAIN, LOG_TAG, "Media Duration: %{public}lld us", duration);

Demuxer Core: OH_AVDemuxer Instance Management

Key references:

Header: <multimedia/player_framework/native_avdemuxer.h> Library: libnative_media_avdemuxer.so System Capability:

SystemCapability.Multimedia.Media.Spliter

1. Creation and Lifecycle Control

Demuxer instance must be created from a Source instance. Strict destruction order: destroy Demuxer first, then Source.

// 1. Create Demuxer instance
mDemuxer = OH_AVDemuxer_CreateWithSource(mSource);
if (!mDemuxer) {
    OH_AVSource_Destroy(mSource);
    return -1;
}

// ... playback logic ...

// 2. Destroy instance (same instance can only be destroyed once)
if (mDemuxer != nullptr) {
    OH_AVDemuxer_Destroy(mDemuxer);
    mDemuxer = nullptr; // Good practice: nullify after destruction
}

2. Track Selection Logic

Iterate tracks to identify video (MEDIA_TYPE_VID) and audio (MEDIA_TYPE_AUD) tracks, then explicitly select them via OH_AVDemuxer_SelectTrackByID.

int32_t trackCount = 0;
OH_AVFormat_GetIntValue(sourceFormat, OH_MD_KEY_TRACK_COUNT, &trackCount);

for (int32_t i = 0; i < trackCount; i++) {
    auto trackFormat = OH_AVSource_GetTrackFormat(mSource, i);
    int32_t trackType = -1;
    OH_AVFormat_GetIntValue(trackFormat, OH_MD_KEY_TRACK_TYPE, &trackType);

    if (trackType == MEDIA_TYPE_VID && mVideoTrackIndex == -1) {
        mVideoTrackIndex = i;
        OH_AVDemuxer_SelectTrackByID(mDemuxer, i); // Critical: tell demuxer we need this track
    } else if (trackType == MEDIA_TYPE_AUD && mAudioTrackIndex == -1) {
        mAudioTrackIndex = i;
        OH_AVDemuxer_SelectTrackByID(mDemuxer, i);
    }
}

Why must SelectTrackByID be called? High-performance players may only need video (e.g., VR social preview) or only audio. Active track selection greatly reduces I/O overhead from parsing irrelevant tracks.

[!NOTE] To switch bitrate or disable a stream during playback, use OH_AVDemuxer_UnselectTrackByID to remove unwanted tracks.

Sample Reading: ReadSampleBuffer in Practice

1. Advantages of OH_AVDemuxer_ReadSampleBuffer

Since API 11, ReadSampleBuffer replaces the older ReadSample. It reads directly into the decoder's OH_AVBuffer, enabling efficient memory reuse and richer metadata transfer.

// Called in decoder callback OnNeedInputParameter
auto ret = OH_AVDemuxer_ReadSampleBuffer(mDemuxer, trackIndex, dataBuffer);
if (ret == AV_ERR_OK) {
    // Success: dataBuffer now contains raw bitstream (e.g., H.265 NALU)
    OH_AVCodecBufferAttr attr;
    OH_AVBuffer_GetBufferAttr(dataBuffer, &attr);
    // attr.pts is the frame timestamp, crucial for synchronization
}

2. DRM Support

Commercial VR content often uses DRM. HarmonyOS demuxer natively supports DRM info callbacks. Recommended API 12+ interface:

// DRM info callback example
void OnMediaKeySystemInfoUpdate(OH_AVDemuxer *demuxer, DRM_MediaKeySystemInfo *info) {
    // Retrieve DRM scheme info and trigger license verification
    OH_LOG_Print(LOG_APP, LOG_INFO, LOG_DOMAIN, LOG_TAG, "DRM info received.");
}

// Set callback during initialization
OH_AVDemuxer_SetDemuxerMediaKeySystemInfoCallback(mDemuxer, OnMediaKeySystemInfoUpdate);

[!IMPORTANT] Prefer SetDemuxerMediaKeySystemInfoCallback . The older SetMediaKeySystemInfoCallback doesn't return the demuxer instance, causing context identification issues in multi-stream parallel scenarios; slated for deprecation in API 14.

Seek Optimization: Instant Response for VR

VR video files are large; slow seek response causes noticeable "freeze" which is more discomforting in VR than 2D.

Seek Strategy: SEEK_MODE_PREVIOUS_SYNC

QVRPlayer uses "seek to previous sync frame" mode.

void QVRDecoder::Seek(int64_t timeMs) {
    if (mDemuxer) {
        // Jump to nearest keyframe (I-frame) before timeMs
        OH_AVDemuxer_SeekToTime(mDemuxer, timeMs, SEEK_MODE_PREVIOUS_SYNC);
    }
}

Technical point: After seek, must clear decoder and render queues of stale buffers to prevent "frame rebound" artifacts.

Performance Optimization: Pushing Demuxing to the Limit

For 4K/8K VR, the following enhancements are applied:

1. Multi-threaded Parallel Pre-reading : Demuxing runs off UI thread. A dedicated C++ pre-read thread loads next batch of data into buffers while decoder processes current frame.

2. M3U8 Index Acceleration : For network streams, OH_AVSource auto-parses HLS indexes. Proper NDK-layer network cache configuration enables ultra-fast channel switching.

Summary

This chapter explored QVRPlayer's "shell cracker" — OH_AVDemuxer. By building efficient track discovery and sample reading mechanisms, we've established the data pathway for subsequent hardcore hardware decoding.

Next chapter: "HarmonyOS 6 VR Player Development 4: Hardware Decoder (AVCodec) Efficient Hard Decoding & Surface Mode Practice" — tackling the toughest challenge: driving HarmonyOS hardware codec chips to transform raw bitstreams into stunning panoramic pixels.

Technical Map & API Index

[!TIP] Development Context: Header: <multimedia/player_framework/native_avdemuxer.h> Library: libnative_media_avdemuxer.so Capability: SystemCapability.Multimedia.Media.Spliter

[!TIP] Core API Overview: OH_AVDemuxer_CreateWithSource : Bind data source. OH_AVDemuxer_SelectTrackByID : Enable media track. OH_AVDemuxer_ReadSampleBuffer : Get compressed frame (API 11+). OH_AVDemuxer_SeekToTime : Timeline precision seek. OH_AVDemuxer_SetDemuxerMediaKeySystemInfoCallback : Listen DRM info (API 12+). OH_AVDemuxer_Destroy : Safe resource release.