Turning Images into Video with Seamless OpenGL Transitions on Android

1. Introduction

Image‑to‑video composition with PPT‑like transitions is a common feature in short‑video apps such as Douyin; it involves three parts: image‑to‑video synthesis, timeline definition, and OpenGL effects.

Two Processing Schemes

First, pre‑compose the video from images without processing, record timestamps, and apply OpenGL effects during the transition period.

Second, process images and apply effects while writing frames to the video.

Scheme 1 is modular but requires duplicate processing; Scheme 2 interleaves processing and effect generation, reducing time, so Scheme 2 is chosen.

2. Image Composition

Common techniques:

FFMPEG : Define output format and frame rate, then list images to generate a video.

ffmpeg -r 1/5 -i img%03d.png -c:v libx264 -vf fps=25 -pix_fmt yuv420p out.mp4

MediaCodec : Decode and encode video frames; write image data as YUV with timestamps.

MediaCodec + OpenGL : Combine MediaCodec pipeline with OpenGL to generate transition effects while encoding.

Technical Implementation

A clock controls the frame write frequency and encoder timestamps, e.g., 24 fps requires a 42 ms interval per frame. An EGL environment is created to invoke OpenGL; the surface from MediaCodec is bound to an EGLSurface.

Images are decoded to Bitmaps, uploaded as textures via texImage2D, and the Y‑axis is flipped to match Android’s canvas coordinate system.

int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
int textureId = textures[0];
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureId);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);

When loading the second image, a non‑zero texture unit is activated to handle multiple textures. GLES20.glActiveTexture(GLES20.GL_TEXTURE1); During rendering, blending is enabled for opacity transitions, and vertex/texture coordinates are bound before drawing.

GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
GLES20.glUseProgram(mProgramHandle);
GLES20.glEnableVertexAttribArray(positionHandle);
GLES20.glEnableVertexAttribArray(texCoordHandle);
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texName);
GLES20.glUniform1i(samplerHandle, 0);
GLES20.glEnable(GLES20.GL_BLEND);
GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA);
GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
GLES20.glDisableVertexAttribArray(positionHandle);
GLES20.glDisableVertexAttribArray(texCoordHandle);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);

3. Transition Timeline

For an image list {1,2,3,4}, each stage defines display time showT and animation time animT. The first and last stages simply show images; intermediate stages include both images and transition timestamps. The timeline ensures enterEndT = exitStartT for seamless continuity.