How PaddleDepth and Paddle3D Enable Low‑Cost 3D Vision Development

Background and Application Scenarios

3D vision has become a hot research area because many real‑world tasks, such as intelligent manufacturing, autonomous systems, and medical imaging, require accurate three‑dimensional information. Traditional 2D vision focuses on color images, while 3D tasks need depth data for reconstruction and analysis.

Challenges in 3D Vision Data Acquisition

Existing depth‑sensing devices are expensive, produce sparse point clouds (LiDAR) or low‑resolution depth maps (ToF cameras), and often limit practical deployment. Reducing hardware cost while maintaining dense, high‑quality depth information is a key obstacle.

PaddleDepth: Low‑Cost Depth Information Suite

PaddleDepth addresses the above problems by offering three core capabilities:

Depth Super‑Resolution – enhances low‑resolution depth maps to obtain denser 3D reconstructions.

Depth Completion – fills missing regions in sparse LiDAR point clouds, producing dense depth estimates.

Depth Estimation from RGB – directly predicts depth from a single color image, dramatically lowering sensor costs.

The toolkit includes more than ten state‑of‑the‑art models and four novel self‑developed algorithms, all achieving SOTA performance on public benchmarks. On the KITTI dataset, PaddleDepth ranks first in self‑supervised monocular depth estimation, supervised binocular depth estimation, and depth completion. On Middlebury, it leads the depth super‑resolution task and won the ECCV2020 Robust Vision Challenge stereo matching competition.

Paddle3D: End‑to‑End 3D Perception Development Kit

Paddle3D provides a full pipeline for 3D perception, from data preparation to model deployment. Its architecture consists of a framework layer built on PaddlePaddle, a toolbox layer with dataset integrations and 3D operators, an algorithm layer containing a rich model zoo, and a tool layer that integrates with the Apollo autonomous driving platform.

The model library covers:

Monocular 3D detection (e.g., SMOKE, CaDDN) – low‑cost camera‑only solutions.

LiDAR‑based point‑cloud detection (e.g., PointPillars, IA‑SSD) – high‑precision 3D bounding boxes.

Multi‑modal fusion models – combine camera and LiDAR data for robustness.

Multi‑view detectors (e.g., BEVFormer, PETR) – state‑of‑the‑art performance on bird‑eye‑view tasks.

To mitigate the high memory and compute demands of point‑cloud networks, Paddle3D adopts sparse convolution (SparseConv) which skips invalid calculations, reducing both memory usage and FLOPs. Models such as PV‑RCNN and Voxel R‑CNN leverage this capability.

Training Example

train_dataset = KittiMonoDataset(
    dataset_root='datasets/KITTI', mode='train',
    transforms=[
        T.LoadImage(reader='pillow', to_chw=False),
        T.Gt2SmokeTarget(mode='train', num_classes=3),
        T.Normalize(mean=[0.485, 0.456, 0.406],
                    std=[0.229, 0.224, 0.225])
    ])

model = SMOKE(
    backbone=DLA34(),
    head=SMOKEPredictor(num_classes=3),
    depth_ref=[28.01, 16.32],
    dim_ref=[[3.88, 1.63, 1.53], [1.78, 1.70, 0.58], [0.88, 1.73, 0.67]])

lr_scheduler = paddle.optimizer.lr.MultiStepDecay(
    milestones=[36000, 55000], learning_rate=1.25e-4)

optimizer = paddle.optimizer.Adam(
    learning_rate=lr_scheduler, parameters=model.parameters())

trainer = Trainer(model=model, optimizer=optimizer, iters=20, train_dataset=train_dataset)
trainer.train()

Configuration Example

batch_size: 8
iters: 70000

train_dataset:
  type: KittiMonoDataset
  dataset_root: datasets/KITTI
  transforms:
    - type: LoadImage
      reader: pillow
      to_chw: False
    - type: Normalize
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]

lr_scheduler:
  type: MultiStepDecay
  milestones: [36000, 55000]
  learning_rate: 1.25e-4

optimizer:
  type: Adam

Running the training script is as simple as:

python tools/train.py --config configs/smoke/smoke_dla34_no_dcn_kitti.yml --iters 20 --log_interval 1 --num_worker 5

Conclusion

PaddleDepth provides a cost‑effective solution for acquiring high‑quality depth data through completion, super‑resolution, and RGB‑based estimation, while Paddle3D delivers a comprehensive, modular, and Apollo‑compatible framework for 3D perception research and deployment. Together they lower entry barriers and accelerate innovation in 3D vision applications.