Technical Exploration of JD.Vision on Apple Vision Pro: Spatial Computing, Custom Gestures, and Rendering Optimizations

Overview

JD.Vision is one of the first native applications on Apple Vision Pro, leveraging spatial computing to provide an immersive shopping experience where users can drag 1:1 scaled 3D products into their real environment for realistic preview.

Key Technologies

The platform utilizes Video See‑Through (VST) with a 12 ms latency, eye‑tracking for high‑precision hand‑eye control, and the broader concept of space‑computing that digitizes the real world to enable interactive 3D experiences.

Implementation Details

Development uses VisionOS containers (Windows, Volumes, Spaces) and RealityView for dynamic 3D model loading. Coordinate transformations between image, camera, and world spaces are handled via SwiftUI's CoordinateSpaceProtocol and RealityCoordinateSpace. Custom gesture recognition extends Apple’s basic gestures by feeding hand‑tracking key points into rule‑based, DNN, and LSTM modules, enabling richer interactions. Custom shaders built with Shadergraph provide specialized material effects and UI grids without distortion.

Optimization

To manage the higher data dimensionality of spatial computing, the team applies dynamic model quality adjustment, Reality Composer Pro packaging, resource pre‑loading, caching, and efficient collision detection to support multi‑product placement and maintain smooth performance.

Future Directions

Future work includes expanding to full‑category shopping, adding high‑quality 3D assets, depth video, and advanced features such as 3D scene search, intelligent recommendation, and virtual try‑on, further enhancing the immersive retail experience.