How JD.Vision Leverages Apple Vision Pro’s Spatial Computing for Immersive Shopping

Introduction

Apple Vision Pro officially launched in China on June 28, and JD.Vision became one of the first native applications on the platform, offering an immersive shopping experience based on spatial computing. Users can download the app from the visionOS App Store and initially explore home appliances, furniture, and trendy digital products, with plans to expand to the full catalog.

What Makes Vision Pro Different?

Vision Pro features Video See‑Through (VST) with a latency of only 12 ms, far lower than competing headsets, and integrates high‑precision eye‑tracking for a hand‑eye control system that surpasses traditional controller accuracy. Its spatial‑computing concept digitizes the real environment and enables interactive 3D experiences.

3D Product and Scene Display

Vision Pro provides three content containers—Windows, Volumes, and Spaces. JD.Vision uses a Windows container for the home screen, embedding both 2D and 3D elements, and employs RealityView to load dynamic 3D models on a static homepage, creating a virtual product showcase.

Virtual‑Physical Fusion with Spatial Computing

Vision Pro combines cameras, LiDAR, ambient‑light sensors, and the powerful M2/R1 chips to achieve high‑precision spatial mapping. Using Apple’s ARKitScenes as an example, the app detects planes, builds a 3D map of the real environment, and anchors virtual items so they appear correctly sized and oriented in the user’s space.

Custom Gesture Recognition

Beyond Apple’s six built‑in gestures, the team captured 25 hand‑key points per hand and fed them into rule‑based systems, DNNs, and LSTMs to recognize additional dynamic gestures for precise rotation and scaling of 3D products.

Custom Shaders

Leveraging the M2 and R1 chips, custom shaders created with Shadergraph render special material effects such as post‑collision mesh highlights, breathing lights on products, and a UI nine‑grid that stays undistorted across different model sizes.

Spatial Computing Optimization

Processing 3D data adds a dimension of computation. The team dynamically adjusts model polygon counts per product category, packs scenes with Reality Composer Pro, and uses pre‑loading, dynamic loading, and caching to keep the experience fluid.

Future Exploration

JD plans to expand the immersive catalog with more high‑quality 3D models, depth videos, 3D scene search, intelligent recommendation, and virtual try‑on, continuously refining the mixed‑reality shopping experience as Vision Pro matures.

References

Andrei, Constantin‑Octavian. “3D affine coordinate transformations.” (2006).

A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion.

Dynamic Hand Gesture Recognition Based on Short‑Term Sampling Neural Networks.

https://www.cnblogs.com/ghjnwk/p/10852264.html

https://developer.mozilla.org/en-US/docs/Games/Techniques/3D_collision_detection

https://developer.apple.com/documentation/realitykit/

https://github.com/apple/ARKitScenes

https://developer.apple.com/documentation/arkit/arkit_in_ios/configuration_objects/understanding_world_tracking