JointLight: Precise AI Relighting for Human Videos via Asynchronous RGB-Irradiance Modeling

Research Background

Video relighting requires understanding the interplay between light sources, geometry, materials, and occlusion: changing illumination should alter shading and shadows on humans and scenes while preserving appearance and motion. This capability is key for video models to express physical world dynamics. Traditional rendering relies on accurate 3D geometry and material data, which are hard to obtain from ordinary video. Diffusion models lower the barrier: IC-Light demonstrated generative lighting editing but struggles with precise light-direction and shadow control. DiffusionRenderer and UniRelight support environment-map-driven relighting yet still implicitly infer light-scene interaction, leading to inaccuracies under complex occlusion and motion, and limited generalization from simple synthetic training. DiLightNet and PractiLight introduce coarse geometry-based lighting cues, but depth errors and boundary gaps cause shading artifacts that propagate into results; per-frame geometry errors also risk temporal flicker. JointLight addresses this by using geometry to provide useful shading cues while allowing the model to correct unreliable details.

Algorithm Principle

Geometry Prior for Irradiance Cues

JointLight first answers "where should be bright or dark." It records directional light color and intensity via an HDR environment map. From the input video, it estimates per-frame monocular point clouds, converts them to meshes, and removes triangles crossing depth discontinuities. A uniform white diffuse material is assigned, and Mitsuba 3 renders the mesh under the target environment light, producing a coarse irradiance map aligned with the video frame. However, this coarse irradiance contains artifacts: fine details like hair, cloth folds, and foliage are lost; character edges show black gaps; local brightness deviates from ground truth (Figure 2). Directly using such maps as fixed conditions would inject errors — black edges, false shadows, detail loss — and cause flicker in video. Therefore, JointLight treats coarse irradiance as a correctable cue, using ground-truth irradiance as the training target.

Asynchronous RGB-Irradiance Joint Modeling

JointLight fine-tunes the Wan2.2-TI2V-5B video generation model. The input video, noisy RGB video, and noisy irradiance video are fed into a single diffusion transformer. The model simultaneously generates the relit frames and refines the irradiance distribution; the two modalities correspond spatially and temporally and exchange information within the same model. "Asynchronous" means RGB and irradiance can be at different denoising timesteps: during training, separate noise timesteps are sampled for each modality, teaching the model to cooperate even when one modality is less clear. Thus irradiance need not be a perfectly reliable condition image; it participates in generation and is progressively corrected.

Simply adding coarse irradiance as a condition fails to accurately reproduce target lighting; joint modeling alone cannot fully bridge the gap between coarse cues and true irradiance. To adapt to real video, a domain-matching technique is added: in a second training stage, noisy coarse cues randomly replace noisy ground truth, exposing the model to inference-time geometry errors. Added noise attenuates high-frequency artifacts while preserving large-scale shading; joint modeling lets the model supplement and correct local details using input content and target illumination.

At inference, noisy irradiance guides from the start but is initially held fixed while the RGB branch denoises; after a preset timestep, both denoise jointly. This leverages the low-frequency lighting distribution from geometry-based coarse irradiance while mitigating high-frequency flaws via noise.

High-Quality Synthetic Data for Sim2Real Transfer

The team used Unreal Engine 5 to fix scene, character, and motion while varying only lighting, rendering paired RGB and ground-truth irradiance — paired data difficult to capture in real shoots. Realistic imagery and diverse scenes narrow the synthetic-real gap. The dataset comprises 6,800 dynamic scenes, 68,000 videos, covering 30 indoor/outdoor environments, 60 characters, 600 body animations, 20 expressions, and 821 HDR environment maps. Each scene uses 10 lighting conditions, 20% with rotating lights. This diversity teaches the model a wide range of light-scene relationships. Based on this pipeline, two test benchmarks — SynthStatic (fixed target lighting) and SynthDynamic (rotating target lighting) — were built with paired ground truth for evaluating lighting control accuracy and generation quality.

Results

Static Lighting: Balancing Lighting Accuracy and Appearance Preservation

Relighting must respond to target illumination while preserving identity, clothing texture, and scene content. DiffusionRenderer and UniRelight respond better to light direction and color but still exhibit skin/cloth texture distortion and local shading inaccuracies in complex scenes. JointLight, by explicitly encoding shading relationships via irradiance cues, adjusts illumination while better preserving original appearance. On SynthStatic, it achieves PSNR 16.62 dB and LPIPS 0.251 (best among compared methods); SSIM 0.656 is slightly below UniRelight's 0.661. On the unseen real-world HumanOLAT dataset, JointLight scores PSNR 23.89 dB, SSIM 0.928, LPIPS 0.078 — all best among compared methods.

Dynamic Lighting: Continuous Adjustment with Moving Light Sources

When human motion and light changes occur simultaneously, the model must continuously update shading direction, shadows, and local brightness. Experiments rotate the target environment light to test coherent response. On SynthDynamic, JointLight reaches PSNR 16.39 dB, SSIM 0.631, LPIPS 0.263 — all three metrics outperform compared methods. Relative to UniRelight, PSNR improves by 1.49 dB and LPIPS decreases by 0.040. Video comparisons confirm that explicit irradiance guidance helps maintain control as target lighting varies over time.

Additional Results

Irradiance serves both as a lighting control basis and a window into how the model adjusts scene shading. Through asynchronous joint modeling of pixels and irradiance, the model produces fine-grained irradiance maps alongside relit videos. Explicit lighting control grounded in geometric priors yields more accurate, physically plausible shadows, especially under directional sources like sunlight.

Conclusion

Video generation is shifting from visual realism toward controllable expression of scene physics. Lighting bridges 3D scenes and 2D imagery, revealing shape and material while determining how humans and environments visually integrate. To let generative models change shading per creator intent, the relationships among light sources, geometry, and occlusion must be embedded in generation. JointLight constructs shading cues from geometric priors, employs asynchronous RGB-irradiance joint modeling and domain matching to let the model correct lighting cues while generating frames, combining the interpretability of physical rendering with the expressiveness of video generation. Given an input video and target environment light, JointLight reorganizes shading and shadows of humans and surroundings, preserving post-capture creative freedom for lighting adjustment. This offers a flexible lighting editing tool for film post-production, digital human presentation, and virtual content creation, and demonstrates the value of geometric priors in generative modeling: even imperfect 3D information can effectively constrain generation and guide physical relationships. As geometry, materials, and light transport are more deeply integrated into generative processes, video models will support richer scene editing, extending creator control from pixel appearance to the physical factors that form the image.

References

[1] Henglei Lv, Bailin Deng, Xiaoqiang Liu, Yichen Wang, Haoxian Zhang, Pengfei Wan, Lin Gao*. JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling. SIGGRAPH Asia, 2026.

[2] Lvmin Zhang, Anyi Rao, Maneesh Agrawala. Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport. ICLR, 2025.

[3] Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, Zian Wang. DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR, 2025: 26069–26080.

[4] Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang. UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting. arXiv:2506.15673, 2025.

[5] Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, Xin Tong. DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation. SIGGRAPH, 2024: 1–12.

[6] Yotam Erel, Rishabh Dabral, Vladislav Golyanik, Amit H. Bermano, Christian Theobalt. PractiLight: Practical Light Control Using Foundational Diffusion Models. ACM TOG, 44(6), 2025: 1–11.

[7] Team Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[8] Timo Teufel, Pulkit Gera, Xilong Zhou, Umar Iqbal, Pramod Rao, Jan Kautz, Vladislav Golyanik, Christian Theobalt. HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis. ICCV, 2025: 29131–29141.

[9] Wenzel Jakob et al. Mitsuba 3 renderer. https://mitsuba-renderer.org.

[10] Unreal Engine 5. https://www.unrealengine.com/unreal-engine-5.