Skip to content
JointLight: Controllable Human Video Relighting via Asynchronous RGB-Irradiance Joint Modeling

JointLight: Controllable Human Video Relighting via Asynchronous RGB-Irradiance Joint Modeling

JointLight, from UCAS, Cardiff University, and Kling AI, enables precise relighting of human-centric videos by fusing geometric priors with asynchronous RGB-irradiance joint modeling, achieving state-of-the-art results on static and dynamic lighting benchmarks while generalizing to real-world footage.

Data Party THU
Data Party THU
Data Party THU

Research Background

World models aim to let machines understand how environments change and predict the results of interventions, supporting agent learning, simulation, and content creation. This requires video models to maintain consistent physical relationships during scene changes. Lighting is a key link: when light sources move, people, objects, and shadows should respond coherently. Controllable relighting thus becomes an entry point for exploring the physical expression capabilities of video models.

Relighting demands that a model understand the relationships among light sources, geometry, materials, and occlusion: after illumination changes, the brightness and shadows of people and scenes should change accordingly while preserving original appearance and motion. This is also an important capability for video models to express the physical world. Traditional rendering relies on accurate geometry and material information, but ordinary video cannot provide complete 3D structure, and reconstruction and capture of complex human scenes are costly.

Diffusion models lower the barrier for relighting. Methods like IC-Light show the potential of generative lighting editing, but relying only on text or environment-map conditions makes precise control of light direction and shadows difficult. DiffusionRenderer and UniRelight further support environment-map-driven relighting, yet still require the network to implicitly infer how light interacts with the scene; faced with complex occlusion and human motion, local shading relationships may deviate from the target illumination, and simpler synthetic training scenes limit generalization to real scenes.

DiLightNet, PractiLight, and similar methods introduce coarse geometry-rendered lighting cues, but depth errors and boundary gaps cause shading deviations; directly using them as control conditions may propagate artifacts into results. Video also requires cross-frame lighting consistency, and per-frame geometry errors further increase flickering risk. The key is to exploit the valid shading information provided by geometry while allowing the model to correct unreliable details—this is the starting point of JointLight.

Method: JointLight

Geometric Priors Provide Shading Cues

Lighting control first needs to answer "where should be bright, where dark." JointLight uses HDR environment maps to record the color and intensity of light from all directions, then estimates monocular point clouds per frame from the input video and converts them to meshes, removing triangles that cross depth discontinuities. A uniform white diffuse material is assigned to the mesh, and Mitsuba 3 renders it under the target environment light, yielding a coarse irradiance map aligned with the frame. However, the constructed coarse irradiance contains many flaws; using it directly as a fixed lighting condition easily introduces errors.

As shown in Figure 2, compared to ground-truth irradiance, the coarse irradiance loses detail in hair, clothing folds, and foliage; black gaps appear at human boundaries, and local brightness deviates. If these results are used as fixed conditions, the model may bake gaps and erroneous shading into the generated frames, creating black edges, fake shadows, or detail loss; in video, per-frame geometry changes can cause lighting flicker. Therefore, JointLight treats coarse irradiance as a correctable cue, using ground-truth irradiance as the generation target during training.

Figure 2: Comparison of RGB, ground-truth irradiance, and coarse irradiance. Geometry errors cause coarse irradiance to deviate from true shading relationships.
Figure 2: Comparison of RGB, ground-truth irradiance, and coarse irradiance. Geometry errors cause coarse irradiance to deviate from true shading relationships.

Figure 2: Comparison of RGB, ground-truth irradiance, and coarse irradiance. Geometry errors cause coarse irradiance to deviate from true shading relationships.

Joint Generation of Video and Irradiance

JointLight fine-tunes the Wan2.2-TI2V-5B video generation model. The input video, noisy RGB video, and noisy irradiance video are fed into the same diffusion Transformer. The model generates the relit frames while refining the shading distribution; the two modalities correspond spatially and temporally and exchange information within the same model.

Figure 3: Joint training pipeline. RGB and irradiance use independent noise timesteps and exchange information in the same model.
Figure 3: Joint training pipeline. RGB and irradiance use independent noise timesteps and exchange information in the same model.

Figure 3: Joint training pipeline. RGB and irradiance use independent noise timesteps and exchange information in the same model.

The term "asynchronous" means RGB and irradiance can be at different denoising progress. During training, separate noise timesteps are sampled for each modality, teaching the model to cooperate even when frame and shading cue clarity differ. Thus, irradiance need not be a perfectly reliable condition map; it can participate in generation and be progressively corrected.

Directly using coarse irradiance as a condition fails to accurately reproduce target lighting; merely adding RGB-irradiance joint modeling is insufficient to close the gap between coarse cues and true irradiance. To adapt to real video, training incorporates domain matching: in a second stage, noisy coarse cues randomly replace noisy ground truth, exposing the model to inference-time geometry errors early. Adding appropriate noise attenuates high-frequency artifacts while preserving large-scale shading relationships; joint modeling lets the model supplement and correct local details based on input content and target illumination.

Figure 4: Inference pipeline. Geometry estimation, irradiance rendering, noise addition, then asynchronous sampling produces relighting results.
Figure 4: Inference pipeline. Geometry estimation, irradiance rendering, noise addition, then asynchronous sampling produces relighting results.

Figure 4: Inference pipeline. Geometry estimation, irradiance rendering, noise addition, then asynchronous sampling produces relighting results.

At inference, noisy irradiance participates from the start but is initially held fixed while the RGB branch denoises; after a preset timestep, both denoise together. This lets the model effectively exploit the low-frequency lighting distribution from the geometry-based coarse irradiance, while noise addition mitigates the negative impact of high-frequency flaws.

High-Quality Synthetic Data for Sim2Real Transfer

The team used Unreal Engine 5 to fix scenes, characters, and motions while varying only lighting, synchronously rendering RGB and ground-truth irradiance to obtain paired data difficult to capture in real shoots. Realistic imagery and diverse scenes help bridge the synthetic-real gap.

The dataset contains 6,800 dynamic scenes and 68,000 videos, covering 30 indoor/outdoor environments, 60 characters, 600 body animations, 20 expressions, and 821 HDR environment maps. Each scene uses 10 lighting conditions, 20% with rotating lights. Diverse combinations let the model learn more lighting relationships.

Based on this pipeline, the authors also built SynthStatic and SynthDynamic test benchmarks, using fixed and rotating target environment lights respectively, providing paired ground truth for dynamic scene relighting to evaluate lighting control accuracy and generation quality.

Figure 5: Diverse synthetic scenes provide paired RGB and irradiance, supporting transfer to real video.
Figure 5: Diverse synthetic scenes provide paired RGB and irradiance, supporting transfer to real video.

Figure 5: Diverse synthetic scenes provide paired RGB and irradiance, supporting transfer to real video.

Results

Static Lighting: Balancing Lighting Accuracy and Appearance Preservation

Relighting must respond to target illumination while preserving human identity, clothing texture, and scene content. DiffusionRenderer and UniRelight better respond to light direction and color, but still suffer from skin/clothing texture distortion or inaccurate local shading in complex scenes. JointLight uses irradiance cues to explicitly establish shading relationships in the scene, adjusting illumination while better preserving original appearance. On SynthStatic, it achieves PSNR 16.62 dB, LPIPS 0.251 (best among compared methods); SSIM 0.656, slightly below UniRelight's 0.661.

Table 1: HumanOLAT static image relighting results.
Table 1: HumanOLAT static image relighting results.

Table 1: HumanOLAT static image relighting results.

To further test transfer, evaluation on the unseen HumanOLAT real-captured dataset yields PSNR 23.89 dB, SSIM 0.928, LPIPS 0.078—all best among compared methods.

Dynamic Lighting: Continuous Adjustment with Target Illumination

When human motion and light source changes occur simultaneously, the model must continuously update shading direction, shadows, and local brightness, increasing control difficulty. Experiments rotate the target environment light to test whether generated results respond coherently to illumination changes. On SynthDynamic, JointLight achieves PSNR 16.39 dB, SSIM 0.631, LPIPS 0.263—all three metrics outperform compared methods. Compared to UniRelight, PSNR improves by 1.49 dB and LPIPS decreases by 0.040. Combined with video comparisons, this shows explicit irradiance guidance helps the model maintain control when target lighting varies over time.

Table 2: SynthDynamic dynamic lighting test results.
Table 2: SynthDynamic dynamic lighting test results.

Table 2: SynthDynamic dynamic lighting test results.

Additional Results

Irradiance provides both lighting control basis and a way to observe how the model adjusts scene shading distribution. Through asynchronous joint modeling of pixels and irradiance, the model produces fine-grained irradiance maps alongside relit videos.

Figure 8: Shading illustration. From left: input, relighting result, irradiance map, target environment light.
Figure 8: Shading illustration. From left: input, relighting result, irradiance map, target environment light.

Figure 8: Shading illustration. From left: input, relighting result, irradiance map, target environment light.

Explicit lighting control fused with geometric priors makes video shading more precisely controllable and physically plausible, especially under direct light sources like the sun.

Figure 9: More results.
Figure 9: More results.

Figure 9: More results.

Conclusion

Video generation is moving from pursuing visual realism toward controllable expression of physical scene relationships. Lighting is a crucial bridge connecting 3D scenes to 2D imagery: it reveals object shape and material, and determines how people and environments visually integrate. To let generative models change shading according to creator intent, the relationships among light sources, geometry, and occlusion must be incorporated into the generation process. JointLight constructs shading cues from geometric priors, employs asynchronous RGB-irradiance joint modeling and domain adaptation, enabling the model to correct lighting cues while generating frames—combining the interpretability of physical rendering with the expressiveness of video generation, providing a path for precise control of shading in dynamic scenes.

Given an input video and target environment light, JointLight can reorganize the brightness and shadows of people and surrounding scenes, preserving creative lighting adjustment space after capture. This offers more flexible shading editing for film post-production, digital human presentation, and virtual content creation, and demonstrates the value of geometric priors in generative modeling: even imperfect 3D information can serve as effective constraints to guide physical relationships. As geometry, materials, and light transport are more deeply integrated into generation, video models are expected to support richer scene editing, extending creator control from image appearance to the physical factors that form the image.

References

[1] Henglei Lv, Bailin Deng, Xiaoqiang Liu, Yichen Wang, Haoxian Zhang, Pengfei Wan, Lin Gao*. JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling. SIGGRAPH Asia, 2026.

[2] Lvmin Zhang, Anyi Rao, Maneesh Agrawala. Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport. ICLR, 2025.

[3] Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, Zian Wang. DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR, 2025: 26069–26080.

[4] Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang. UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting. arXiv:2506.15673, 2025.

[5] Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, Xin Tong. DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation. SIGGRAPH, 2024: 1–12.

[6] Yotam Erel, Rishabh Dabral, Vladislav Golyanik, Amit H. Bermano, Christian Theobalt. PractiLight: Practical Light Control Using Foundational Diffusion Models. ACM TOG, 44(6), 2025: 1–11.

[7] Team Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[8] Timo Teufel, Pulkit Gera, Xilong Zhou, Umar Iqbal, Pramod Rao, Jan Kautz, Vladislav Golyanik, Christian Theobalt. HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis. ICCV, 2025: 29131–29141.

[9] Wenzel Jakob et al. Mitsuba 3 renderer. https://mitsuba-renderer.org.

[10] Unreal Engine 5. https://www.unrealengine.com/unreal-engine-5.

Resources

Paper Title: JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling

Project Page: http://www.geometrylearning.com/JointLight

Code Repository: https://github.com/IGLICT/JointLight

Code example

来源:机器之心
本文
约4000字
,建议阅读
5
分钟
基于异步RAG与辐射度建模融合的可控光照编辑。
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

diffusion modelsasynchronous modelinggeometric priorshuman-centric videoSIGGRAPH Asia 2026video relightingcontrollable lightingirradiance estimation
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.