iPhone LiDAR Scan + AI = Walkable Three.js Room in Browser: 2.6k Stars in 13 Days

The GitHub project 3D Model Anything (by Wes Bos) gained 2.6k stars in 13 days. It enables users to scan a room with an iPhone Pro (LiDAR + camera), export the data, and then use an AI-assisted workflow to rebuild the space as a Three.js scene that can be explored in a browser — first-person navigation, wireframe view, and a "knolling" exploded view are included.

Four-step workflow

Scan: Walk around the room with the iPhone. LiDAR measures distances; the camera captures photos. The app shows real-time coverage (red = missing, green = sufficient). The author's office scan produced 165 photos, 1,112 depth frames, and 501,651 triangles.

Export: A ZIP file is shared via system share sheet (AirDrop, etc.). No cloud lock-in; data stays local.

AI reconstruction: The repo includes a scan-to-threejs skill pack — a detailed instruction set for an AI coding assistant. The assistant reads the measurement data and photos, builds geometry piece by piece (walls, desk, chair, lamp), then compares each rendered view against the original photos, iterating until they match. The skill pack forces the AI to label every part as "measured", "modeled", or "estimated" and forbids assuming two scans with the same room name are the same space (to avoid drift errors).

View in browser: The output is a npm run dev Three.js app at http://localhost:5188. Features: orbit controls, first-person walk, wireframe toggle, knolling view (all objects laid out flat).

Key technical details

Wall plane error: 2.6 cm (good for furniture placement, not for construction drawings).

68% of mesh points align with wall surfaces.

Data package is fully open: manifest, room data, meshes, photos, depth frames, with validation scripts.

Furniture in the scan is only bounding boxes (e.g., 1.86 m × 0.83 m); geometry, materials, and details are inferred by AI from photos — the most labor-intensive step.

Limitations and gotchas

Slow by design: The author's office took dozens of AI–human comparison rounds; solo users face hundreds of iterations.

Hard alignment cannot fix drift or moved objects.

Known errors in v4: A stair-step wall behind a lounge chair was misclassified as an opening; an acoustic panel was labeled a window (photos 84–89 prove no opening).

Environment setup: Requires Python 3, Pillow, NumPy, Open3D.

iOS app not on App Store: Must compile with Xcode (needs Mac, iOS 17+, LiDAR-equipped iPhone/iPad Pro).

Demo site currently stuck on "Loading Wes's office"; local npm run dev also hangs without errors.

Mac required: No pre-built Windows binary yet (first issue requests one).

Setup commands

git clone https://github.com/wesbos/3d-model-anything.git
cd 3d-model-anything
npm ci
npm run dev

Open http://localhost:5188 for the viewer.

To use your own room, build the iOS app:

cd apps/house-capture
xcodegen generate
xcodebuild -project HouseCapture.xcodeproj -scheme HouseCapture \
  -configuration Release -destination 'generic/platform=iOS' \
  -derivedDataPath build -allowProvisioningUpdates build

Sign in Xcode, install on device, scan, share via AirDrop, unzip to data/captures-in/<room-name>/, then prompt the AI:

用 scan-to-threejs 重建 data/captures-in/my-room 这个房间，
检查测量数据和照片，建一个可编辑的场景，
把渲染出来的视角跟原始照片对比

Author's take

The novelty isn't scanning (Apple's RoomPlan exists) but the structured AI workflow : build from data, verify against photos, label confidence. Value lies in:

Space designers: rapid editable 3D drafts.

Web developers: ready-made "walk into my space" embed.

Archival: turn physical spaces into durable records.

Still toy-grade: of 374 repo files, most are AI skill docs; only the iOS app source, Three.js runtime, and one demo room are directly usable. Recommend starting with a single, sparse room. The confidence-labeling pattern is transferable to other projects.

Open source: github.com/wesbos/3d-model-anything Demo video and live demo (currently broken):

wesbos.github.io/3d-model-anything