FreeOcc: The First Training‑Free Open‑Vocabulary 3D Occupancy Mapping System (RSS‑2026)

Semantic occupancy prediction, which assigns each voxel a state (free, occupied, unknown) and a semantic label, is essential for embodied robot perception. Existing methods rely heavily on large‑scale 3D occupancy and semantic annotations as well as accurate camera trajectories, limiting their applicability in novel, unlabelled environments.

FreeOcc: A Training‑Free Open‑Vocabulary Solution

The authors propose FreeOcc, the first system that performs open‑vocabulary 3D occupancy prediction without any task‑specific training. It processes monocular or RGB‑D image streams, builds a globally consistent map, and supports arbitrary natural‑language queries.

System Architecture

FreeOcc decomposes online mapping into four modular layers:

Point‑cloud map: A SLAM front‑end (DROID‑SLAM by default, with MASt3R‑SLAM and VGGT‑SLAM evaluated) estimates camera poses and reconstructs a semi‑dense point cloud.

3D Gaussian (3DGS) map: The point cloud anchors a set of 3D Gaussians that are continuously refined. To bridge the gap between rendering‑oriented 3DGS and voxel‑level occupancy, the authors introduce Geometry‑aware Initialization (G‑ini) and Geometrically Anchored Gaussian Updates (GAGU), which keep Gaussian centers fixed to SLAM points and expand them anisotropically along the viewing ray.

Semantic map: A pretrained open‑vocabulary vision‑language model extracts language‑aligned features from each image. These features are lifted to the 3D Gaussian primitives, creating language‑embedded Gaussians.

Occupancy map: A probabilistic Gaussian‑to‑Occupancy projection aggregates nearby Gaussians for each voxel, yielding both a geometric occupancy probability and an open‑vocabulary semantic score.

Key Challenges Addressed

High cost of 3D annotation: FreeOcc eliminates the need for dense occupancy and semantic labels.

Poor cross‑environment generalization: By avoiding dataset‑specific training, the system retains performance on unseen scenes.

Mismatch between rendering‑focused 3DGS and voxel‑level occupancy: G‑ini and GAGU enforce geometric consistency suitable for space reasoning.

Lack of unified evaluation for open‑world occupancy: The authors construct the ReplicaOcc benchmark, a fine‑grained, open‑vocabulary dataset derived from Replica scenes.

Experimental Evaluation

FreeOcc is evaluated on multiple fronts:

EmbodiedOcc‑ScanNet: Without any occupancy or semantic supervision, the monocular version reaches 31.29 IoU / 13.86 mIoU and the RGB‑D version 34.40 IoU / 15.84 mIoU, more than twice the scores of self‑supervised baselines GaussianOcc and GaussTR.

ReplicaOcc benchmark: Zero‑shot transfer shows learning‑based methods collapse (semantic mIoU ≈ 0), while FreeOcc maintains 46.81 IoU / 16.93 mIoU (monocular) and 55.65 IoU / 20.90 mIoU (RGB‑D).

Geometric consistency: Converting various 3DGS‑SLAM outputs to occupancy reveals FreeOcc’s superior average IoU (monocular 39.34, RGB‑D 45.24) over methods such as Photo‑SLAM, MonoGS, DROID‑Splat, SplaTAM, GS‑ICP, and RTG‑SLAM.

Ablation study: Removing GAGU and G‑ini drops performance to 27.98 IoU / 11.20 mIoU (8.8 FPS). Adding GAGU improves to 40.18 IoU / 16.03 mIoU (25 FPS), and the full system reaches 45.03 IoU / 18.37 mIoU (24.6 FPS), demonstrating the importance of both components.

Open‑vocabulary queries: On ReplicaOcc, FreeOcc achieves 31.06 mIoU for the top‑10 categories and retains reasonable scores (23.02, 16.57, 12.01 mIoU) up to the top‑40 categories.

Real‑World Deployment

FreeOcc runs on an Intel i9‑14900KF + RTX 5090 platform, ingesting live RGB‑D streams from an Intel RealSense D435i without pre‑recorded trajectories or pose ground truth. In indoor and outdoor scenes, the system continuously updates the multi‑layer map, and a Qwen3‑VL multimodal model supplies on‑the‑fly object labels, enabling fine‑grained queries such as “red cup” or “blue cup”.

Implications and Future Work

By decoupling occupancy prediction from large‑scale annotation and training, FreeOcc opens a pathway for robots to achieve persistent, language‑driven scene understanding in completely new environments. The authors envision downstream applications in navigation, manipulation, and human‑robot interaction, where natural‑language queries can be answered directly from the continuously built semantic map.