How to Decode and Play Video with MediaCodec on Android Using TextureView

In the previous article we mentioned that playing video on Android is simple with MediaPlayer+SurfaceView or VideoView. This article focuses on the workflow of decoding video with MediaCodec and rendering it on a TextureView.

Video

1. TextureView Create a TextureView to obtain a SurfaceTexture that MediaCodec will use as the output surface for video data.

mMovieView = (TextureView) findViewById(R.id.movie_display);
mMovieView.setSurfaceTextureListener(this);

Initialize the decoder after the SurfaceTexture becomes available, typically in onSurfaceTextureAvailable.

@Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
    // Get SurfaceTexture for decoder output. Decode after it is ready.
    mSurfaceTexture = surface;
    // Start decoding
    decode();
}

2. Obtain video parameters Use MediaExtractor to parse the video file and retrieve format, track index, width, and height.

mMediaExtractor = new MediaExtractor();
mMediaExtractor.setDataSource("/sdcard/test.mp4");
int trackCount = mMediaExtractor.getTrackCount();
for (int i = 0; i < trackCount; i++) {
    MediaFormat format = mMediaExtractor.getTrackFormat(i);
    String mime = format.getString(MediaFormat.KEY_MIME);
    if (mime.startsWith("video/avc")) {
        mVideoFormat = format;
        mVideoTrackIndex = i;
        mMimeType = mime;
        break;
    }
}
int width = mVideoFormat.getInteger(MediaFormat.KEY_WIDTH);
int height = mVideoFormat.getInteger(MediaFormat.KEY_HEIGHT);

3. Start decoding Select the video track and let MediaCodec decode it. mMediaExtractor.selectTrack(mVideoTrackIndex); MediaCodec works asynchronously: it provides an input buffer, you fill it with data from MediaExtractor, then queue it. When decoding finishes, an output buffer becomes available.

- onInputBufferAvailable(MediaCodec mc, int inputBufferId)

This callback indicates an input buffer is ready.

ByteBuffer decoderInputBuffer = mDecoder.getInputBuffer(inputBufferId);

Read data from MediaExtractor and queue it.

mDecoder.queueInputBuffer(inputBufferId, 0, size, presentationTime, mMediaExtractor.getSampleFlags());

If no more data is available, signal end‑of‑stream.

mDecoder.queueInputBuffer(inputBufferId, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);

- onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo info)

When a decoded frame is ready, release it to the SurfaceTexture. If the end of the stream is reached, seek to the beginning and reset the decoder.

boolean reset = ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0);
if (reset) {
    mMediaExtractor.seekTo(0, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
    mDecoder.flush();
    mDecoder.start();
}

Video frame display

Typical video runs at 30 fps (≈33 ms per frame), but actual frame intervals are defined by PTS (Presentation Time Stamp). DTS (Decoding Time Stamp) tells when to decode a frame, while PTS tells when to present it.

During playback, compare the current system time with the frame’s PTS and sleep until the appropriate moment.

long currentPos = info.presentationTimeUs;
if (mOnlinePrevMonoUsec == 0) {
    mOnlinePrevMonoUsec = System.nanoTime() / 1000;
    mOnlinePrevPresentUsec = info.presentationTimeUs;
} else {
    long delta = (info.presentationTimeUs - mOnlinePrevPresentUsec);
    if (delta < 0) delta = 0;
    long desiredUsec = mOnlinePrevMonoUsec + (long) (delta / mSpeed);
    long nowUsec = System.nanoTime() / 1000;
    while (nowUsec < (desiredUsec - 100)) {
        long sleepTimeUsec = desiredUsec - nowUsec;
        if (sleepTimeUsec > 500000) sleepTimeUsec = 500000;
        if (sleepTimeUsec > 0) {
            try {
                Thread.sleep(sleepTimeUsec / 1000, (int) (sleepTimeUsec % 1000) * 1000);
            } catch (InterruptedException ignored) {}
        }
        nowUsec = System.nanoTime() / 1000;
    }
    mOnlinePrevMonoUsec += (long) (delta / mSpeed);
    mOnlinePrevPresentUsec += delta;
}

The complete demo code for video playback is shown below.

package demo.mediacodec.xueting.com.mediacodecdemo;
import android.graphics.SurfaceTexture;
import android.media.*;
import android.os.*;
import android.support.annotation.*;
import android.support.v4.app.FragmentActivity;
import android.util.Log;
import android.view.Surface;
import android.view.TextureView;
import android.widget.LinearLayout;
import java.io.IOException;
import java.nio.ByteBuffer;

public class Mp4PlayerActivity extends FragmentActivity implements TextureView.SurfaceTextureListener {
    private TextureView mMovieView;
    private SurfaceTexture mSurfaceTexture;
    private MediaExtractor mMediaExtractor;
    private MediaCodec mDecoder;
    private String mMimeType;
    private MediaFormat mVideoFormat;
    private int mVideoTrackIndex = -1;
    private int mWidth = 720;
    private int mHeight = 1280;
    // ... onCreate, initExtractor, decode, callbacks, onDestroy as described above ...
}

Audio

The previous code only decodes the video track. A typical video file also contains an audio track, which can be decoded similarly and played back with AudioTrack.

1. Initialize AudioTrack Create an AudioTrack using the channel count and sample rate obtained from MediaExtractor.

private void prepareAudio(int channelCount, int sampleRate) {
    int bufferSize = AudioTrack.getMinBufferSize(sampleRate, AudioFormat.CHANNEL_OUT_STEREO, AudioFormat.ENCODING_PCM_16BIT);
    int channelConfig = channelCount == 2 ? AudioFormat.CHANNEL_OUT_STEREO : AudioFormat.CHANNEL_OUT_MONO;
    mSonic = new Sonic(sampleRate, channelCount);
    mAudioTrack = new AudioTrack(AudioManager.STREAM_MUSIC, sampleRate, channelConfig,
            AudioFormat.ENCODING_PCM_16BIT, bufferSize * 2, AudioTrack.MODE_STREAM);
    mAudioTrack.setStereoVolume(mMute ? 0 : 1, mMute ? 0 : 1);
    mAudioTrack.play();
}

2. Play decoded audio data When the decoder outputs audio chunks, feed them to Sonic for speed/pitch control, then write to AudioTrack.

if (chunk != null) {
    if (BuildConfig.DEBUG) {
        Log.d(TAG, "onBufferAvailable:" + chunk.length);
    }
    // Adjust speed with Sonic
    mSonic.setSpeed(mSpeed);
    mSonic.putBytes(chunk, chunk.length);
    int available = mSonic.availableBytes();
    if (available > 0) {
        byte[] modifiedSamples = new byte[available];
        mSonic.receiveBytes(modifiedSamples, available);
        mAudioTrack.write(modifiedSamples, 0, available);
    }
}