仅用粗略阴影提示,即可实现无需训练的可控物体光照重渲染。
SpotLight: Shadow-Guided Object Relighting via Diffusion
- 通过输入目标阴影提示,驱动预训练扩散模型生成对应光照效果。
- 在无额外训练下,显著提升物体合成的视觉一致性与真实感。
- 适合需要快速定制光照的创意设计、图像编辑用户。
近期工作表明,扩散模型可作为强大的神经渲染引擎,用于将虚拟物体插入图像。然而,与典型物理渲染器不同,这些神经渲染引擎缺乏对光照的手动控制,而光照控制对于改善或个性化图像结果至关重要。本文提出SpotLight,仅需提供物体的粗略阴影提示,即可在不进行任何额外训练的情况下实现精确且可控的光照重渲染。具体而言,将期望的物体阴影注入预训练的扩散神经渲染器,即可准确生成符合目标光源位置的阴影,并使物体及其阴影与背景图像自然融合。本方法完全无需训练,充分利用现有神经渲染技术实现可控光照重渲染。用户研究表明,SpotLight在定量和感知评价上均优于专门设计用于光照重渲染的现有扩散模型。我们还展示了手绘阴影、全图重渲染等扩展应用,验证了其多功能性。
原文摘要 · Abstract (English)
Recent work has shown that diffusion models can serve as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. However, unlike typical physics-based renderers, these neural rendering engines are limited by the lack of manual control over the lighting, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise and controllable lighting can be achieved without any additional training, simply by supplying a coarse shadow hint for the object. Indeed, we show that injecting only the desired shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, is entirely training-free and leverages existing neural rendering approaches to achieve controllable relighting. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting. We also demonstrate other applications, such as hand-scribbling shadows and full-image relighting, demonstrating its versatility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。