Mirage一键生成逼真连贯的驾驶场景视频编辑结果。
Mirage: One-Step Video Diffusion for Photorealistic and Coherent Asset Editing in Driving Scenes
- 用文本到视频扩散模型保证帧间一致性,结合2D预训练编码器注入细节。
- 在多个编辑任务中实现高保真度与时间连贯性,视觉质量优于现有方法。
- 适合自动驾驶数据增强、视频生成研究者使用,代码开源可复现。
以视觉为中心的自动驾驶系统依赖多样且可扩展的训练数据以实现稳健性能。虽然视频对象编辑为数据增强提供了可行路径,但现有方法常难以同时保持高视觉保真度与时间连贯性。本文提出 extbf{Mirage},一种用于驾驶场景中逼真且连贯资产编辑的一步式视频扩散模型。Mirage 基于文本到视频扩散先验以确保跨帧的时间一致性。然而,3D因果变分自编码器常因压缩导致空间保真度下降,直接将3D编码特征传递至解码层会破坏时间因果性。为此,我们从预训练2D编码器注入时序无关潜变量至3D解码器,以恢复细节并保留因果结构。此外,由于场景物体与插入资产在不同目标下优化,其高斯分布存在错配,导致姿态对齐偏差。为此,我们引入两阶段数据对齐策略,结合粗粒度3D对齐与细粒度2D精修,从而改善对齐效果并提供更清晰的监督。大量实验表明,Mirage 在多种编辑场景中均实现了高真实感与时间连贯性。除资产编辑外,Mirage 还可泛化至其他视频到视频翻译任务,可作为未来研究的可靠基线。代码已公开于 https://github.com/wm-research/mirage。
原文摘要 · Abstract (English)
Vision-centric autonomous driving systems rely on diverse and scalable training data to achieve robust performance. While video object editing offers a promising path for data augmentation, existing methods often struggle to maintain both high visual fidelity and temporal coherence. In this work, we propose \textbf{Mirage}, a one-step video diffusion model for photorealistic and coherent asset editing in driving scenes. Mirage builds upon a text-to-video diffusion prior to ensure temporal consistency across frames. However, 3D causal variational autoencoders often suffer from degraded spatial fidelity due to compression, and directly passing 3D encoder features to decoder layers breaks temporal causality. To address this, we inject temporally agnostic latents from a pretrained 2D encoder into the 3D decoder to restore detail while preserving causal structures. Furthermore, because scene objects and inserted assets are optimized under different objectives, their Gaussians exhibit a distribution mismatch that leads to pose misalignment. To mitigate this, we introduce a two-stage data alignment strategy combining coarse 3D alignment and fine 2D refinement, thereby improving alignment and providing cleaner supervision. Extensive experiments demonstrate that Mirage achieves high realism and temporal consistency across diverse editing scenarios. Beyond asset editing, Mirage can also generalize to other video-to-video translation tasks, serving as a reliable baseline for future research. Our code is available at https://github.com/wm-research/mirage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。