How We Built a 3D Immersive Food Marketplace with Babylon and React

Project Overview

The "Taste Adventure" project provides a mobile‑first 3D shopping experience where users control an avatar with a joystick to explore virtual food venues. The main technical goal is to integrate Babylon.js 3D rendering with React DOM UI while keeping the app in portrait orientation.

Rendering Implementation

3D Scene Rendering & Orientation Handling

Because the host app does not support native landscape mode, the entire UI is rotated 90° in portrait. The CSS below creates a fixed‑size wrapper that swaps width/height and applies a rotation transform. When the device reports landscape, the transform is removed.

.wrapper{position:fixed;width:100vh;height:100vw;top:0;left:0;transform-origin:left top;transform:rotate(90deg) translateY(-100%);transform-style:preserve-3d;}@media only screen and (orientation:landscape){.wrapper{width:100vw;height:100vh;transform:none;}}

Resource Management Center

Large numbers of meshes and textures are loaded through a centralized manager that de‑duplicates assets and uses Babylon’s addMeshTask / addTextureTask. The manager loads assets sequentially on mobile to limit memory pressure.

async appendMeshAsync(tasks, withLoading = true) {
  const promiseList = [];
  const uniqTasks = _.uniqWith(tasks, (a, b) => a.name === b.name);
  for (const item of uniqTasks) {
    const {name, rootUrl, fileName, modelRoot = ''} = item;
    if (this.modelAssets.has(name)) { console.log(`${name} already loaded`); continue; }
    const promise = new Promise((resolve, reject) => {
      const task = this.assertManager.addMeshTask(`${Tools.RandomId}_task`, modelRoot, rootUrl, fileName);
      task.onSuccess = result => { this.savemodelAssets(name, result); resolve(result); };
      task.onError = () => reject(null);
    });
    promiseList.push(promise);
  }
  this.assertManager.loadAsync();
  return await Promise.all(promiseList);
}

DOM Components

Standard e‑commerce UI (product list, detail page, prize pop‑ups) is built with React. Horizontal scrolling on the rotated UI is implemented by hiding overflow, measuring touch distance on the axis perpendicular to the intended scroll direction, and translating the list accordingly.

Product List

Data is fetched from a backend configuration and rendered with name, image, price and promotion tags.

Product Detail

Reuses the generic product component and adds an "Add to Cart" flow.

3D Model Popup

When a collectible item is selected, a separate WebGL canvas renders the 3D model while the main scene is paused. The render loop is stopped/started based on the popup visibility.

componentDidUpdate(prevProps) {
  const prevShow = prevProps.stampDetailModal.isShow;
  const curShow = this.props.stampDetailModal.isShow;
  if (prevShow !== curShow) {
    this.engine.stopRenderLoop();
    if (!curShow && !this.engineRunningStatus) {
      this.engine.runRenderLoop(() => this.mainScene.render());
    }
    this.engineRunningStatus = curShow;
  }
}

Mixed Mode Rendering

Both Babylon and React render simultaneously. UI elements that are cheap to render (buttons, lists) stay in the DOM, while 3D models stay in Babylon. Interaction is mediated by an event center.

Example: when a product is selected, Babylon captures a screenshot, darkens it, and sets it as the background of a UI layer. The 3D model then receives input, which triggers a DOM update to fetch product details.

Tools.CreateScreenshot(this.scene.getEngine(), activeCamera, {width, height}, data => {
  setProductBg(data);
});
function setProductBg(pic) {
  const ui = this.UIList.get('productUI');
  if (ui && ui.layer) {
    const img = new Image('bg', pic);
    img.width = '100%'; img.height = '100%';
    ui.addControl(img);
    const mask = new Rectangle('mask');
    mask.width = '100%'; mask.height = '100%'; mask.thickness = 0;
    mask.background = 'rgba(0,0,0,0.5)';
    ui.addControl(mask);
    ui.layer.layerMask = detailLaymask;
  }
}

Camera Collision (Air Wall)

Global collisions are enabled and any mesh whose ID starts with "wall" is marked as invisible and collidable. When the camera gets stuck, a ray is cast from the avatar toward the camera; the nearest intersection point moves the camera to a safe position.

// Enable global collisions
this.scene.collisionsEnabled = true;
// Mark wall meshes as collidable and invisible
if (mesh.id.toLowerCase().indexOf('wall') === 0) {
  mesh.visibility = 0;
  mesh.checkCollisions = true;
  obstacle.push(mesh);
}

Map Function

The map displays the avatar’s current location. Opening the map triggers an event that writes the character’s position and rotation into a store, which then renders a marker.

export function toggleShow(isShow) {
  eventCenter.getInstance().trigger(EVENT_TYPES.GET_CHARACTER_POSITION);
  store.dispatch({type: EActionTypes.TOGGLE_SHOW, payload: {isShow}});
}
this.appEventCenter.on(EVENT_TYPES.GET_CHARACTER_POSITION, () => {
  const pos = this.character?.position ?? {x:0, z:0};
  const rot = this.character?.mesh.rotation.y ?? 0;
  updatePos({x: -pos.x, y: pos.z, rotation: rot - Math.PI});
});

New‑User Guidance

A racing‑game‑style guide line points users to target buildings. The visible length is the projection of the vector from avatar to target onto the guide direction, computed with a dot product.

setProgress(val) {
  if (typeof val === 'number') {
    this.progress = val;
  } else if (this.startPoint && this.endPoint) {
    const startToEnd = this.endPoint.add(this.startPoint.negate());
    const dir = Vector3.Normalize(startToEnd);
    this.progress = 1 - Vector3.Dot(dir, this.endPoint.add(val.negate())) / startToEnd.length();
  }
}

The guide line is rendered with a custom shader that provides gradient opacity and animation.

# Vertex shader
uniform mat4 worldViewProjection;
uniform vec2 uScale;
attribute vec4 position;
attribute vec2 uv;
varying vec2 st;
void main() {
  gl_Position = worldViewProjection * position;
  st = vec2(uv.x * uScale.x, uv.y * uScale.y);
}
# Fragment shader
uniform sampler2D textureSampler;
uniform vec2 uScale;
uniform float uOffset;
uniform float uAlphaTransStart;
uniform float uAlphaTransEnd;
varying vec2 st;
void main() {
  vec2 rst = vec2(st.x, -st.y + uOffset);
  float alphaEnd = smoothstep(0., uAlphaTransEnd, st.y);
  float alphaStart = smoothstep(uScale.y, uScale.y - uAlphaTransStart, st.y);
  float alpha = alphaStart * alphaEnd;
  gl_FragColor = vec4(texture2D(textureSampler, rst).rgb, alpha);
}

Performance Optimizations

Texture resolution limited to ≤ 1K × 1K (some 512 × 512) to reduce memory.

Textures converted to GPU‑only compressed formats (PVRTC / ASTC) so they do not occupy JavaScript heap.

High‑poly detail baked into textures and applied to low‑poly meshes, preserving visual fidelity while keeping mesh vertex count low.

Other Pitfalls

Lighting : Adding an HDR environment map and a directional light dramatically improved brightness and reflections.

GUI Rendering Clarity : Babylon’s built‑in GUI layer produced blurry text on high‑DPI screens. Switching UI elements to React DOM restored crispness without extra GPU load.

Emissive Materials : Babylon lacked proper emissive shading. The effect was baked into textures and approximated with simple lights.

References

Early demo: https://jelly.jd.com/article/623acd52f25db001d3f9d1fa

Precise mesh intersection discussion: https://forum.babylonjs.com/t/precise-mesh-intersection-detection/8444

Portrait‑vs‑landscape handling article: http://www.woshipm.com/pd/4459087.html