arXiv:2503.03689cs.CV2025-03被引 7

用双分支扩散模型生成高保真驾驶场景视频,提升细节与一致性。

DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance

  • 引入体素射线采样作为条件输入,融合语义与3D几何信息。
  • 在NuScenes上FID降低4.09%,目标检测mAP提升1.46%。
  • 适合自动驾驶场景生成、多视图视频合成的研究者使用。

精准且高保真的驾驶场景重建需要有效利用全面的场景信息作为条件输入。现有方法主要依赖3D边界框和BEV道路地图控制前景与背景,难以捕捉驾驶场景的全部复杂性,也未能充分整合多模态信息。本文提出DualDiff,一种双分支条件扩散模型,用于增强多视角及视频序列中的驾驶场景生成。具体而言,我们引入体素射线采样(ORS)作为条件输入,提供丰富的前景与背景语义以及3D空间几何信息,精确控制两类元素的生成。为提升复杂且远距离前景物体的合成质量,提出前景感知掩码(FGM)去噪损失函数。同时,设计语义融合注意力(SFA)机制,动态优先处理相关信息并抑制噪声,实现更有效的多模态融合。最后,为确保图像到视频生成的质量,引入奖励引导扩散(RGD)框架,保持生成视频的全局一致性和语义连贯性。大量实验表明,DualDiff在多个数据集上达到最先进性能:在NuScenes数据集上,相比最佳基线FID降低4.09%;在下游任务中,车辆的mIoU提升4.50%,道路mIoU提升1.70%,前景mAP提升1.46%。代码将开源于https://github.com/yangzhaojason/DualDiff。

原文摘要 · Abstract (English)

Accurate and high-fidelity driving scene reconstruction demands the effective utilization of comprehensive scene information as conditional inputs. Existing methods predominantly rely on 3D bounding boxes and BEV road maps for foreground and background control, which fail to capture the full complexity of driving scenes and adequately integrate multimodal information. In this work, we present DualDiff, a dual-branch conditional diffusion model designed to enhance driving scene generation across multiple views and video sequences. Specifically, we introduce Occupancy Ray-shape Sampling (ORS) as a conditional input, offering rich foreground and background semantics alongside 3D spatial geometry to precisely control the generation of both elements. To improve the synthesis of fine-grained foreground objects, particularly complex and distant ones, we propose a Foreground-Aware Mask (FGM) denoising loss function. Additionally, we develop the Semantic Fusion Attention (SFA) mechanism to dynamically prioritize relevant information and suppress noise, enabling more effective multimodal fusion. Finally, to ensure high-quality image-to-video generation, we introduce the Reward-Guided Diffusion (RGD) framework, which maintains global consistency and semantic coherence in generated videos. Extensive experiments demonstrate that DualDiff achieves state-of-the-art (SOTA) performance across multiple datasets. On the NuScenes dataset, DualDiff reduces the FID score by 4.09% compared to the best baseline. In downstream tasks, such as BEV segmentation, our method improves vehicle mIoU by 4.50% and road mIoU by 1.70%, while in BEV 3D object detection, the foreground mAP increases by 1.46%. Code will be made available at https://github.com/yangzhaojason/DualDiff.

视频生成扩散模型自动驾驶多模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。