Technical Exploration of JD.Vision Development on Apple Vision Pro: Spatial Computing, Custom Gestures, and Rendering Optimizations

In June of the previous year Apple released its first head‑mounted device, Apple Vision Pro, which launched in China on June 28. JD.Vision, a native application for Vision Pro, initially focuses on home appliances, furniture, and trendy digital products, with plans to expand to the full catalog.

Vision Pro distinguishes itself from earlier headsets through its low‑latency Video See‑Through (VST) technology (12 ms), high‑precision eye‑tracking enabling accurate hand‑eye control, and the broader concept of spatial computing that digitizes the real environment for immersive interaction.

The JD retail technology team describes the technical challenges of building JD.Vision, including adapting to visionOS’s unbounded spatial canvas, handling the lack of mature examples, and extending the platform with custom gestures, collision effects, and component systems.

1. 3D Product and Scene Display – Vision Pro offers three container types (Windows, Volumes, Spaces). JD.Vision uses a Windows container for the home screen and RealityView for dynamic 3D model loading, enabling interactive 2D/3D product showcases.

2. Spatial‑Computing Applications – Leveraging multiple sensors (cameras, LiDAR, ambient light) and Apple’s M2/R1 chips, the app performs high‑precision environment mapping, coordinate‑space transformations, and affine transformations to allow users to freely move, rotate, and scale virtual products within their real space.

3. Custom Gesture Recognition – Beyond Apple’s built‑in tap, pinch, zoom, and rotate gestures, JD.Vision adds AI‑driven custom gestures by feeding 25 hand‑key‑point data into rule‑based, DNN, and LSTM modules, enabling more precise 3D product manipulation.

4. Custom Shaders – Using the power of the M2 and R1 chips, the team creates custom shaders via Composer Shadergraph to render special material effects, UI grids, and collision visualizations without sacrificing performance.

5. Spatial‑Computing Optimization – To handle the increased data dimensionality, JD.Vision dynamically adjusts model quality, employs Reality Composer Pro for asset compression, and implements resource pre‑loading, dynamic loading/unloading, and caching to maintain smooth interactions.

The article concludes with future exploration directions, such as expanding 3D content, integrating depth video, and enhancing immersive shopping features like 3D scene search, intelligent recommendation, and virtual try‑on.

References include academic papers on affine coordinate transformations, hand‑gesture recognition using LSTM, and various Apple developer documentation links.