How 4D Gaussian Splatting Works: From Canonical Gaussians to Deformation Fields

Real‑time rendering of static 3D scenes is already challenging, and dynamic scenes add the difficulty of moving objects, fluids, and deformable geometry. Storing a full set of Gaussian positions, rotations, and scales for every frame would cause linear memory explosion (e.g., 100 M Gaussians × 100 frames).

4D‑GS solves this by discarding per‑frame storage entirely. It maintains a single canonical 3D Gaussian set G and predicts a deformation ΔG with a lightweight neural network, so the rendered scene at any timestamp is G' = G + ΔG.

The deformation field is factorised into six 2‑D planes (XY, XZ, YZ, XT, YT, ZT) following the HexPlane/K‑Planes trick. For a query point (e.g., μ = (1.234, 5.678, 2.0)) at time t, bilinear interpolation on each plane yields four nearest grid vertices; their distance‑weighted average produces six feature vectors. These vectors are combined by element‑wise (Hadamard) multiplication into a dense vector f_h, which is fed to a tiny MLP to obtain the context vector f_d.

HexPlane initialization is crucial: spatial planes are filled with uniform random values in [0.1, 0.5] to give the network room to learn spatial features, while temporal planes are set to 1.0 so that multiplying by the time feature leaves spatial features unchanged, effectively starting the model from a static scene. The network only learns motion when the loss forces it.

The decoder (director) consists of multiple linear heads that translate f_d into concrete physical commands: position offset ΔX, rotation offset Δr, and scale offset Δs. These offsets are added to the canonical Gaussians, and the resulting scene is rasterised with a differentiable splatting renderer. Notably, 4D‑GS deliberately does not predict colour ΔC or opacity Δα; allowing the network to cheat by fading points in and out would break the physical geometry and hinder downstream 3D tracking. The authors note that these predictions can be re‑enabled for fluids or extreme non‑rigid motion.

Training uses a two‑stage schedule. During the first 3000 iterations the deformation field is frozen, forcing the canonical Gaussians to learn an optimal static geometry (static warm‑up). Afterwards the deformation field is activated (4D optimisation) and the network predicts motion, rendering images with a differentiable splatter and computing loss. The loss combines an L1 image term L₁ with a total‑variation term L_{tv} applied directly to the HexPlane grids, encouraging smooth motion.

Back‑propagation follows two gradient paths: Path A updates the canonical Gaussians G, and Path B updates the deformation field parameters. Learning rates are scheduled separately: decoder weights decay from 1.6 × 10⁻⁴ to 1.6 × 10⁻⁵, while HexPlane feature values decay from 1.6 × 10⁻³ to 1.6 × 10⁻⁴. Training runs up to ~20 000 iterations, after which the canonical geometry stabilises and the deformation field accurately reproduces video motion.

Multi‑view data (e.g., 15–20 cameras) provides dense supervision, preventing the model from collapsing geometry to satisfy a single view. Monocular setups rely on temporal continuity in the HexPlane, making them more prone to local minima, especially when the camera itself moves (as in HyperNeRF), where separating camera motion from object motion remains challenging.

In summary:

Deformation replaces per‑frame tracking, keeping memory cost tied to the deformation network rather than video length.

Factorising the 4D space into six 2D planes and using bilinear interpolation avoids the O(N⁴) memory of a full 4D voxel grid.

Constraining the network to only predict geometric offsets (not colour or opacity) forces it to learn true physical motion, at the cost of extra training tricks, but yields representations useful for downstream 3D tracking.