Optimizing Small Video Playback & Recording on iOS: Solutions and Performance Tips

Small video is a major feature of WeChat 6.0. During development many challenges arise. This article first outlines the product requirements, then presents several implementation schemes, analyzes the advantages and disadvantages of each, and finally summarizes the optimal solution.

Small Video Playback Requirements

Multiple videos must be able to play simultaneously.

Videos should continue playing while the user interacts with the UI.

Playback must not block the UI; as soon as a video scrolls into view it should start playing immediately.

Videos in a list play muted; tapping to enlarge plays with sound.

Small Video Playback Solutions

1. MPMoviePlayerController

MPMoviePlayerController is a simple, easy‑to‑use video playback control that can play local files and network streams, supporting formats such as mov, mp4, mpv, 3gp (H.264 and MPEG‑4). It offers drag‑to‑seek, fast‑forward, rewind, pause, fullscreen, and provides playback‑state notifications. Usage involves setting a URL, adding its view to a parent view, and calling play.

Drawbacks: only one MPMoviePlayerController instance can play at a time, which does not satisfy the requirement for simultaneous playback, and it does not support muted playback. It is suitable mainly for fullscreen video scenarios.

2. AVPlayer

AVPlayer, part of the AVFoundation framework, is a lower‑level playback control offering powerful features but a more complex API. AVPlayer itself does not render video; it must be attached to an AVPlayerLayer. It works together with AVPlayerItem, which handles resource loading, playback settings, and state management via KVO.

Example code for creating an AVPlayerItem and observing its status:

NSURL *videoUrl = [NSURL fileURLWithPath:m_path isDirectory:NO];
m_playItem = [AVPlayerItem playerItemWithURL:videoUrl];
[m_playItem addObserver:self forKeyPath:@"status" options:NSKeyValueObservingOptionNew context:nil];

Creating the player and its layer view:

// AVPlayer
m_player = [AVPlayer playerWithPlayerItem:m_playItem];
m_player.actionAtItemEnd = AVPlayerActionAtItemEndNone;
// AVPlayerLayerView (custom UIView)
m_playerView = [[AVPlayerLayerView alloc] initWithFrame:self.bounds];
[self addSubview:m_playerView];
[(AVPlayerLayer *)[m_playerView layer] setPlayer:m_player];
// Observe playback‑ended notification
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(itemPlayEnded:) name:AVPlayerItemDidPlayToEndTimeNotification object:m_playItem];

The status property of AVPlayerItem can be AVPlayerStatusUnknown, AVPlayerStatusReadyToPlay, or AVPlayerStatusFailed. When the status becomes ReadyToPlay, calling play starts playback.

Compared with MPMoviePlayerController, AVPlayer can play up to 16 videos simultaneously. However, it occupies the AudioSession, affecting other audio‑using components (e.g., chat windows). Proper cleanup (setting the player item to nil) is required to avoid lingering decode threads. Performance testing on a chat window with several small videos showed noticeable UI lag during scrolling, likely due to lock contention between video decoders.

3. AVAssetReader + AVAssetReaderTrackOutput (Custom Player)

To avoid AVPlayer’s performance issues, a custom player can be built using AVAssetReader to obtain decoded audio/video data. Combined with AVAssetReaderTrackOutput, it reads frame‑by‑frame CMSampleBufferRef objects, which can be converted to CGImageRef for display.

Sample decoder class (simplified):

AVAssetReader *reader = [[AVAssetReader alloc] initWithAsset:m_asset error:&error];
NSArray *videoTracks = [m_asset tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *videoTrack = [videoTracks objectAtIndex:0];
NSDictionary *options = @{ (id)kCVPixelBufferPixelFormatTypeKey : @(m_pixelFormatType) };
AVAssetReaderTrackOutput *videoReaderOutput = [[AVAssetReaderTrackOutput alloc] initWithTrack:videoTrack outputSettings:options];
[reader addOutput:videoReaderOutput];
[reader startReading];
while ([reader status] == AVAssetReaderStatusReading && videoTrack.nominalFrameRate > 0) {
    CMSampleBufferRef videoBuffer = [videoReaderOutput copyNextSampleBuffer];
    [m_delegate mMovieDecoder:self onNewVideoFrameReady:videoBuffer];
    CFRelease(videoBuffer);
    [NSThread sleepForTimeInterval:sampleInternal];
}
[m_delegate mMovieDecoderOnDecodeFinished:self];

A companion view ( MVideoPlayerView) receives the decoded frames, converts them to CGImageRef, and sets layer.contents. This approach avoids copying image data until the layer is rendered.

Audio decoding is also possible with AVAssetReader, but the author has not yet implemented audio playback from CMSampleBufferRef, resulting in mute‑only video.

4. Solution Comparison

Performance and power‑consumption tests were conducted under two scenarios:

Scrolling test: iPhone 4 chat window with 30 small videos, scrolling back and forth four times.

Power test: iPhone 5s, maximum screen brightness, auto‑lock disabled, flight mode on, three videos playing simultaneously for 10 minutes.

Solution

Scrolling FPS

Power (mHA/s)

AVPlayer

18.7

0.127

MyMoviePlayer (custom AVAssetReader)

42.4

0.110

The custom AVAssetReader solution outperforms AVPlayer in both scrolling smoothness and energy usage. Because it can only play muted video, it is used for chat‑window and timeline playback, while AVPlayer is retained for the sound‑enabled enlarged view.

Small Video Recording Requirements

Support white balance, focus, and zoom.

Record 6‑second clips at 30 fps with minimal frame loss.

Allow recording at various resolutions and bitrates.

Small Video Recording Solution

AVFoundation provides APIs for requirement 1, so the focus here is on requirements 2 and 3.

Recording workflow:

Create an AVCaptureSession and set the desired resolution.

Add AVCaptureInput objects for camera and microphone.

Add AVCaptureOutput objects such as AVCaptureVideoDataOutput and AVCaptureAudioDataOutput. The audio output should be added after session.startRunning to avoid delaying camera start‑up.

Add an AVCaptureVideoPreviewLayer to show the preview UI.

Create a custom MMovieWriter that wraps an AVAssetWriter for file output.

Start capture with startRunning.

Continuously feed video and audio sample buffers from the capture outputs to MMovieWriter, which compresses video resolution and audio bitrate as needed.

Stop capture with stopRunning, finalize the file, and hand it to the upper layer.

On devices newer than iPhone 4 the recording is smooth (30 fps). On iPhone 4 the frame rate drops to 6–8 fps. Removing UI animations adds only 3–4 fps. Profiling revealed that audio‑compression blocks the AVFoundation thread, causing frame loss. Disabling audio compression restores frame rate, but the system camera remains smoother.

Using AVCaptureMovieFileOutput (640×480) yields fluid capture, but the resulting 6‑second file is >2 MB. Re‑encoding with the custom decoder/writer adds 7–8 seconds, slowing down chat video sending.

To solve this, a buffer‑cache layer was introduced: sample buffers from AVCaptureOutput are first cached off the AV thread; a separate writer thread processes the cache when the CPU is idle. This reduces blocking and brings capture performance close to the system camera, with the first second averaging 18 fps and the remainder stabilizing at 30 fps. The post‑capture delay is reduced to about 1 second.

Occasional write‑failure errors occurred (≈6 % of attempts) with OSStatus ‑12633. The error stemmed from a duplicate FrameTime being written. Adding a cache size limit and forcing a write when the limit is reached eliminated the error. The problematic line was commented out:

//m_writer.movieFragmentInterval = CMTimeMakeWithSeconds(1.0, 1000); // AVAssetWriter

Recording Solution Comparison

Tests on iPhone 4 (multiple 6‑second videos, 10 runs) yielded the following averages:

Solution

Average Bitrate (fps/s)

Preview Delay (s)

Without Buffer Cache

7.2

0.64

With Buffer Cache

27.3

1.2

Adding the buffer cache dramatically improves bitrate and overall recording quality, at the cost of a slightly longer preview delay.

References

Apple AVFoundation Programming Guide