DirectFisheye‑GS: A New Breakthrough in Native Fisheye Gaussian Splatting (CVPR 2026)

The paper "DirectFisheye‑GS: Enabling Native Fisheye Input in Gaussian Splatting with Cross‑View Joint Optimization" (CVPR 2026, arXiv:2604.00648) presents a novel framework that integrates a fisheye camera model natively into the 3D Gaussian splatting (3DGS) pipeline. By doing so, the method processes fisheye images without any undistortion preprocessing, preserving full image information.

Technical background : Traditional 3DGS relies on pinhole‑camera multi‑view images, which require many views to cover a scene. Fisheye cameras provide ultra‑wide fields of view, reducing the number of required images, but existing pipelines first undistort the images, causing detail loss and artifacts.

DirectFisheye‑GS framework :

Native fisheye embedding : a differentiable fisheye projection model based on a fourth‑order polynomial with four distortion parameters is inserted into the rendering pipeline. The model’s Jacobian is derived to keep the whole pipeline end‑to‑end differentiable.

Cross‑view joint optimization : all view‑specific Gaussian parameters are optimized simultaneously, preventing the shape distortion and geometric/photometric inconsistency that arise from single‑view random optimization.

Adaptive density control : during training, the importance of each Gaussian is evaluated periodically. Important Gaussians are split to increase local detail density, while redundant ones are pruned, yielding a compact yet accurate representation.

Algorithm pipeline :

Initialization – sparse multi‑view inputs (including fisheye images) are used to create an initial set of 3D Gaussians, each defined by a 3‑D center, a 3‑D covariance matrix, and an RGB color.

Differentiable tile rasterizer – the Gaussians are splatted onto the screen via a differentiable renderer that computes the loss between rendered images and ground‑truth photos.

Parameter optimization – back‑propagation updates the seven parameters of every Gaussian, iteratively reducing the loss.

Adaptive density control – after a fixed number of iterations, Gaussians are split or pruned based on their contribution to the loss.

Experimental validation : All experiments run on a single NVIDIA A100 80 GB GPU. Three public datasets covering small objects (FisheyeNeRF), medium indoor scenes (ScanNet++), and large outdoor environments (Den‑SOFT) are used. Baselines include the original 3DGS, Fisheye‑GS, 3DGUT, and Self‑Cali‑GS. Quantitative metrics (PSNR, SSIM, LPIPS) show that DirectFisheye‑GS matches or exceeds the current state‑of‑the‑art on every dataset, with especially large PSNR gains in extreme‑wide (>60°) fisheye regions. Qualitative results demonstrate removal of edge‑mosaic artifacts, texture blur, and floating Gaussians, while preserving fine detail and cross‑view lighting consistency.

Innovation highlights :

The polynomial fisheye model offers higher flexibility than traditional equidistant models, adapting to complex modern optics.

Derivation of the projection Jacobian enables fully end‑to‑end differentiable training without pre‑processing.

Joint optimization across all views eliminates per‑view distortion and ensures geometric/photometric coherence.

Applications and outlook : The team envisions using 3DGS for e‑commerce product digitization and 4D volumetric live streaming, turning static 2D live‑shopping videos into immersive experiences. Current limitations include the need for multi‑angle capture and long reconstruction times (tens of minutes to hours). Future work will focus on one‑click capture, second‑level reconstruction, and tighter integration with edge‑AI hardware.