Depth Anything: An Open-Source Large-Scale Model for Arbitrary Image Depth Estimation

Dataset Construction

The authors built a data engine that collected ~62 million unlabeled images from the Internet and automatically generated pseudo‑depth labels. Compared with MiDaS v3.1, the approach uses only six labeled datasets (instead of twelve) while discarding NYUv2 and KITTI to ensure zero‑shot evaluation. Some previously used sources (Movies, WSVD) are no longer available, and the RedWeb collection has lower visual quality. Unlabeled data are combined with DINOv2‑pretrained weights for the teacher model, which compensates for the reduced labeled set and improves coverage and robustness.

Self‑Training with Strong Perturbations

A teacher model T is first trained on the labeled set. It then generates dense pseudo‑labels for the unlabeled images, and a student model S is trained on the union of labeled and pseudo‑labeled data. Naïve self‑training yields only marginal gains. The authors hypothesize that when sufficient labeled data are present, additional knowledge from unlabeled data is limited. To force the student to learn extra visual cues, two strong perturbations are applied to each unlabeled image during training:

Strong color distortion, implemented as a combination of color jitter and Gaussian blur.

Strong spatial distortion, realized with CutMix.

These augmentations encourage the student to discover invariant representations and substantially improve the baseline performance on labeled images.

Semantic Auxiliary Supervision via Feature Alignment

Previous attempts that combined semantic segmentation (e.g., RAM + GroundDINO + HQ‑SAM) with depth estimation did not improve accuracy, likely because converting images to discrete class masks discards rich information. Instead, the authors transfer DINOv2’s semantic knowledge to the depth model through an auxiliary feature‑alignment loss. DINOv2’s encoder produces similar embeddings for different parts of the same object; however, depth may vary across those parts. Therefore, the loss excludes pixel pairs whose cosine similarity exceeds a predefined redundancy threshold, allowing the depth model to retain depth‑specific discriminative features while benefiting from semantic guidance.

Training Pipeline

Train a teacher model T on the six labeled datasets.

Use T to generate pseudo‑depth labels for the 62 M unlabeled images.

Train a student model S on the combined labeled + pseudo‑labeled data, applying the strong color and spatial perturbations and the auxiliary feature‑alignment loss.

Experimental Results

https://arxiv.org/abs/2401.10891<br/>https://github.com/LiheYoung/Depth-Anything<br/>https://depth-anything.github.io/

Zero‑shot evaluation on six public depth benchmarks and on randomly captured images shows strong generalization. Fine‑tuning on depth data yields new state‑of‑the‑art performance. Compared with MiDaS v3.1, the proposed method improves both AbsRel and δ₁ on all tested datasets. For example, on DDAD the AbsRel metric improves from 0.251 to 0.230 and δ₁ from 0.766 to 0.789. The ViT‑B variant surpasses MiDaS, and the smaller ViT‑S variant outperforms MiDaS on several unseen datasets. Additional experiments demonstrate that the learned encoder transfers effectively to semantic segmentation tasks.