将渲染视频转化为逼真视频,同时保持3D结构和动态一致
RealMaster: Lifting Rendered Scenes into Photorealistic Video
- 用几何条件引导扩散模型,从首尾帧逐步生成真实感中间帧
- 在GTA-V数据集上显著提升真实感,且完全保留原始3D控制信息
- 支持中途出现的物体角色,推理无需依赖锚点帧,适合影视制作
当前顶尖视频生成模型虽具高度真实感,但缺乏对场景细节的精确控制,且因无显式几何结构而无法保证3D一致性。相反,3D引擎虽具备精细控制与天然3D一致性,输出却常陷入“诡异谷”效应。弥合这一仿真到现实的差距,需兼具结构精度(严格保留输入的几何与动态)与全局语义转换(整体重构材质、光照、纹理以实现真实感)。本文提出RealMaster,利用视频扩散模型将渲染视频提升为逼真视频,同时与3D引擎输出完全对齐。训练时,通过基于锚点的传播策略生成成对数据:首帧与末帧增强真实感,并以几何条件为引导,向中间帧传播。随后在这些成对视频上训练IC-LoRA,将流水线的高质量输出提炼为可泛化的模型,能处理序列中出现的对象与角色,且推理无需锚点帧。在复杂GTA-V序列上的评估表明,RealMaster显著优于现有视频编辑基线,在保持原始3D控制的同时大幅提升了真实感。
原文摘要 · Abstract (English)
State-of-the-art video generation models produce remarkable photorealism, but they lack the precise control required to align generated content with specific scene requirements. Furthermore, without an underlying explicit geometry, these models cannot guarantee 3D consistency. Conversely, 3D engines offer granular control over every scene element and provide native 3D consistency by design, yet their output often remains trapped in the "uncanny valley". Bridging this sim-to-real gap requires both structural precision, where the output must exactly preserve the geometry and dynamics of the input, and global semantic transformation, where materials, lighting, and textures must be holistically transformed to achieve photorealism. We present RealMaster, a method that leverages video diffusion models to lift rendered video into photorealistic video while maintaining full alignment with the output of the 3D engine. To train this model, we generate a paired dataset via an anchor-based propagation strategy, where the first and last frames are enhanced for realism and propagated across the intermediate frames using geometric conditioning cues. We then train an IC-LoRA on these paired videos to distill the high-quality outputs of the pipeline into a model that generalizes beyond the pipeline's constraints, handling objects and characters that appear mid-sequence and enabling inference without requiring anchor frames. Evaluated on complex GTA-V sequences, RealMaster significantly outperforms existing video editing baselines, improving photorealism while preserving the geometry, dynamics, and identity specified by the original 3D control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。