How to Blend Real-World Textures with Virtual Objects Using OpenGL MVP

Introduction

AR technology continuously strives to merge virtual models with the real environment so that the distinction between reality and virtuality disappears, delivering a realistic rendering experience. This article explores the implementation of environment texture rendering that tightly integrates virtual objects with real‑world scenes.

Background

Two AR effects are demonstrated: a realistic ground opening animation and a ground‑digging animation, showing how the real ground can drive virtual model animations.

Step‑by‑Step Implementation

Step 1: Create a Simple Model

A basic cube model is built in Maya. The cube’s top plane is placed below the horizontal plane, and two half‑plane nodes are added to achieve a left‑right opening animation. A video node is inserted between the half‑planes so that when the ground opens, a video plays on the box.

Step 2: Bind Node Texture with Real‑Scene Texture

The camera continuously captures frames, which are rendered onto the model’s plane texture. The core OpenGL code is:

glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, _dstTexture, 0);

Here _dstTexture is the target texture for the left‑right split animation. Rendering the camera frames to this texture allows the virtual object to appear as if it is part of the real scene.

To isolate only the portion of the real scene that corresponds to the cube’s top plane, the texture must be cropped.

Step 3: Crop the Real‑Scene Texture

Accurate cropping must adapt to the model’s rotation, translation, and scaling. The rendering pipeline transforms vertices through several coordinate spaces:

Local Space (Object Space)

World Space

View Space (Eye Space)

Clip Space

Screen Space

These transformations constitute the familiar MVP (Model‑View‑Projection) process.

The five steps are:

Local coordinates describe the object relative to its own origin.

Convert local coordinates to world coordinates.

Transform world coordinates to view coordinates based on the camera.

Project view coordinates into clip space, clamping them to the [-1, 1] range.

Apply the viewport transform to map clip space to screen space, after which rasterization produces fragments.

After MVP transformation and perspective division, the vertex position is in normalized device coordinates. However, texture mapping uses UV coordinates ranging from (0, 0) to (1, 1). Therefore, the position must be converted:

x = position.x/2.0 + 0.5;
y = position.y/2.0 + 0.5;

Combining MVP transformation, perspective division, and the OpenGL‑to‑UV conversion yields the exact UV coordinates needed to crop the environment texture.

Deployment

The environment‑texture rendering technique has been integrated into Alipay’s “AR Star Treasure” (星宝) feature starting from version 10.1.35, enabling realistic ground‑opening effects.

Challenges

When binding environment textures, it is often necessary to freeze the texture updates after a specific MVP is captured, preventing continuous updates that would break the illusion of a static ground. For models with skeletal animation, the animation matrices affect vertex positions before MVP processing. To avoid updating the texture during animation, the original (pre‑animation) vertex positions are used for texture cropping.

Extensions

The same technique can be applied to reflective objects such as virtual spheres, where environment maps are projected onto the sphere’s surface. Apple’s ARKit 2.0 uses environment texturing to generate cubemap probes for realistic reflections in AR scenes.