arXiv:2507.20331cs.CV2025-07ICCV被引 1

融合3D渲染与2D扩散模型,实现动态手腕上真实感手链插入。

From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videos

  • 用3DGS保证时序一致性,再用2D扩散模型优化光照细节。
  • 通过分离材质属性并逐帧优化,提升光影真实感。
  • 适合增强现实、虚拟试戴等动态视频编辑场景。

将3D物体插入视频是计算机图形学中的长期挑战,广泛应用于增强现实、虚拟试穿和视频合成。在动态场景中,保持时间一致性与真实光照尤为困难,涉及复杂运动、视角变化和光照波动。尽管2D扩散模型能生成逼真的图像,但难以维持跨帧时序连贯性;而传统3D渲染方法虽具时空一致性优势,却难以实现逼真光照。本文提出一种混合插入流程,聚焦于动态手腕场景中的手链插入,利用3D高斯泼溅(3DGS)提供初始渲染的时序一致性,并通过2D扩散增强模型优化光照交互以实现真实感。方法引入基于阴影的处理框架,分离物体固有属性(反照率、阴影、反射率),对阴影与sRGB图像进行联合优化。为保障时序连贯性,采用多帧加权调整优化3DGS模型。这是首个协同3D渲染与2D扩散的视频物体插入方法,为真实且一致的视频编辑提供了稳健解决方案。

原文摘要 · Abstract (English)

Inserting 3D objects into videos is a longstanding challenge in computer graphics with applications in augmented reality, virtual try-on, and video composition. Achieving both temporal consistency, or realistic lighting remains difficult, particularly in dynamic scenarios with complex object motion, perspective changes, and varying illumination. While 2D diffusion models have shown promise for producing photorealistic edits, they often struggle with maintaining temporal coherence across frames. Conversely, traditional 3D rendering methods excel in spatial and temporal consistency but fall short in achieving photorealistic lighting. In this work, we propose a hybrid object insertion pipeline that combines the strengths of both paradigms. Specifically, we focus on inserting bracelets into dynamic wrist scenes, leveraging the high temporal consistency of 3D Gaussian Splatting (3DGS) for initial rendering and refining the results using a 2D diffusion-based enhancement model to ensure realistic lighting interactions. Our method introduces a shading-driven pipeline that separates intrinsic object properties (albedo, shading, reflectance) and refines both shading and sRGB images for photorealism. To maintain temporal coherence, we optimize the 3DGS model with multi-frame weighted adjustments. This is the first approach to synergize 3D rendering and 2D diffusion for video object insertion, offering a robust solution for realistic and consistent video editing. Project Page: https://cjeen.github.io/BraceletPaper/

3D插入视频生成扩散模型光照还原

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。