How to Encode and Mux MP4 Video on Android Using MediaCodec and MediaMuxer

The previous article covered video playback using MediaCodec decoding; this article demonstrates using MediaCodec encoding to generate a video synchronously.

The diagram shows the video and audio synthesis flow: audio is captured via AudioRecord, while video encoding follows the same principle as decoding, and this article explains the encoding and MP4 muxing process.

The basic workflow is to feed data to the encoder and write the encoded output to an MP4 file using MediaMuxer, divided into four steps.

1. Initialize Encoder

// Create media format
MediaFormat outputFormat = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
outputFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
outputFormat.setInteger(MediaFormat.KEY_BIT_RATE, 1024 * 1024 * 8);
outputFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
outputFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 2);

MediaCodec encoder = MediaCodec.createEncoderByType(MIME_TYPE);
encoder.configure(outputFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
InputSurface inputSurface = new InputSurface(encoder.createInputSurface());
encoder.start();

The InputSurface class wraps EGL environment initialization; if you feed raw byte[] data directly, the Surface is not required.

Note: When setting MediaFormat , required parameters such as KEY_BIT_RATE and KEY_FRAME_RATE must be provided, otherwise configure will throw IllegalStateException .

2. Fill Data

Data can be supplied to the encoder in two ways:

Directly fill byte[] data (e.g., raw frames or YUV from the camera).

mEncoderInputBuffers = mEncoder.getInputBuffers();

Then dequeue an input buffer, copy the data, and queue it.

int inputbufferIndex = mEncoder.dequeueInputBuffer(TIME_OUT_US);
if (inputbufferIndex > 0) {
    ByteBuffer buffers = mEncoderInputBuffers[inputbufferIndex];
    buffers.clear();
    buffers.limit(yuv420.length);
    buffers.put(yuv420);
    mEncoder.queueInputBuffer(inputbufferIndex, 0, yuv420.length, presentationTimeUs, 0);
} else {
    Log.e(TAG, "encoder inputbuffer not available:" + inputbufferIndex);
}

Configure an input Surface and refresh it with decoded frames.

decoder.releaseOutputBuffer(decoderStatus, doRender);
if (doRender) {
    outputSurface.awaitNewImage();
    outputSurface.drawImage(false);
    inputSurface.setPresentationTime(info.presentationTimeUs * 1000);
    inputSurface.swapBuffers();
}

The key point is that the decoder’s Surface and the encoder’s Surface exchange data via EGL.

In most cases using a Surface is preferred for performance and to avoid handling different color formats.

3. Encode and Mux (MediaCodec + MediaMuxer)

After encoding, the data is in video/avc format; MediaMuxer packages the video and audio streams into an MP4 container.

MediaMuxer

MediaMuxer facilitates muxing elementary streams. Currently MediaMuxer supports MP4, Webm and 3GP file as the output. It also supports muxing B‑frames in MP4 since Android Nougat.

Typical usage:

MediaMuxer muxer = new MediaMuxer("temp.mp4", OutputFormat.MUXER_OUTPUT_MPEG_4);
MediaFormat audioFormat = new MediaFormat(...);
MediaFormat videoFormat = new MediaFormat(...);
int audioTrackIndex = muxer.addTrack(audioFormat);
int videoTrackIndex = muxer.addTrack(videoFormat);
ByteBuffer inputBuffer = ByteBuffer.allocate(bufferSize);
boolean finished = false;
BufferInfo bufferInfo = new BufferInfo();

muxer.start();
while(!finished) {
    finished = getInputBuffer(inputBuffer, isAudioSample, bufferInfo);
    if (!finished) {
        int currentTrackIndex = isAudioSample ? audioTrackIndex : videoTrackIndex;
        muxer.writeSampleData(currentTrackIndex, inputBuffer, bufferInfo);
    }
}
muxer.stop();
muxer.release();

Support for containers and features varies by Android version (details omitted).

Encoder

After data is fed, retrieve encoded output and write it to the muxer.

int encoderStatus = encoder.dequeueOutputBuffer(info, TIMEOUT_USEC);

MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED – update output buffers.

encoderOutputBuffers = encoder.getOutputBuffers();

MediaCodec.INFO_OUTPUT_FORMAT_CHANGED – obtain the new format and start the muxer.

MediaFormat newFormat = encoder.getOutputFormat();
MediaMuxer mMuxer = new MediaMuxer("/sdcard/test.mp4", MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
mEncodeTrackIndex = mMuxer.addTrack(newFormat);
mMuxer.start();

encoderStatus > 0 – write encoded data.

ByteBuffer encodedData = encoderOutputBuffers[encoderStatus];
if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
    info.size = 0;
}
if (info.size != 0) {
    encodedData.position(info.offset);
    encodedData.limit(info.offset + info.size);
    mMuxer.writeSampleData(mEncodeTrackIndex, encodedData, info);
}
encoder.releaseOutputBuffer(encoderStatus, false);

Note: After the last frame, call stop() on the muxer; otherwise the file will be invalid.

The complete source code is shown below.

public class Mp4Converter {
    private int mWidth = 720;
    private int mHeight = 1280;
    private static final boolean VERBOSE = true;
    private static final String TAG = "Mp4Converter";
    private static final String MIME_TYPE = "video/avc"; // H.264 Advanced Video Coding
    private MediaExtractor mMediaExtractor;
    private MediaFormat mVideoFormat;
    private int mDecodeTrackIndex = -1;
    private int mEncodeTrackIndex = -1;
    private String mMimeType;
    private int mRotation;
    private Surface mOutputSurface;
    private MediaMuxer mMuxer;
    private boolean mMuxerStarted;
    private static final long TIME_OUT_US = 10000;

    public void convert(String source, String target) {
        extractor(source);
        MediaCodec encoder = null;
        MediaCodec decoder = null;
        InputSurface inputSurface = null;
        CodecOutputSurface outputSurface = null;
        try {
            File outputFile = new File(target);
            if (outputFile.exists()) {
                outputFile.delete();
            }
            mMuxer = new MediaMuxer(target, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
            mEncodeTrackIndex = -1;
            mMuxerStarted = false;

            MediaCodecInfo codecInfo = selectCodec(MIME_TYPE);
            if (codecInfo == null) {
                Log.e(TAG, "Unable to find an appropriate codec for " + MIME_TYPE);
                return;
            }
            if (VERBOSE) Log.d(TAG, "found codec: " + codecInfo.getName());
            MediaFormat outputFormat = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
            outputFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
            outputFormat.setInteger(MediaFormat.KEY_BIT_RATE, 1048576 * 3);
            outputFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
            outputFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 2);

            encoder = MediaCodec.createEncoderByType(MIME_TYPE);
            encoder.configure(outputFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
            inputSurface = new InputSurface(encoder.createInputSurface());
            encoder.start();
            decoder = MediaCodec.createDecoderByType(MIME_TYPE);
            outputSurface = new CodecOutputSurface();
            decoder.configure(mVideoFormat, outputSurface.getSurface(), null, 0);
            decoder.start();
            decodeEncodeFromSurfaceToSurface(encoder, inputSurface, decoder, outputSurface);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (inputSurface != null) inputSurface.release();
            if (outputSurface != null) outputSurface.release();
            if (encoder != null) { encoder.stop(); encoder.release(); }
            if (decoder != null) { decoder.stop(); decoder.release(); }
            if (mMuxer != null) { mMuxer.stop(); mMuxer.release(); mMuxer = null; }
        }
    }

    private static MediaCodecInfo selectCodec(String mimeType) {
        int numCodecs = MediaCodecList.getCodecCount();
        for (int i = 0; i < numCodecs; i++) {
            MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);
            if (!codecInfo.isEncoder()) continue;
            String[] types = codecInfo.getSupportedTypes();
            for (String type : types) {
                if (type.equalsIgnoreCase(mimeType)) return codecInfo;
            }
        }
        return null;
    }

    // ... (remaining methods omitted for brevity) ...
}

Refer to the previous article "Android Video Processing – MediaCodec-4 – Video Frame to Image" for the CodecOutputSurface.java implementation.