用视觉大模型特征统一连接仿真与真实驾驶视频生成，解决真实感与控制一致性难题。
Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving
- 利用DINO视觉基础模型特征作为跨域桥梁，融合高层语义与细粒度结构。
- 通过主子空间投影和随机通道尾部丢弃，平衡真实感与生成一致性。
- 引入可学习空间对齐与因果时序聚合，提升高分辨率控制与运动稳定性。
受可控视频扩散模型兴起推动，现有自动驾驶仿真到真实视频生成方法通常依赖显式中间表示来弥合域间差距。然而，这些模态面临根本的「一致性-真实感困境」：低层信号（如边缘、模糊图像）虽能实现精确控制，但会固化合成伪影而降低真实感；高层先验（如深度、语义、HDMaps）虽有助于生成逼真图像，却缺乏保证一致性的结构细节。本文提出Driving with DINO（DwD），一种新框架，利用视觉基础模型（VFM）特征作为仿真与真实世界之间的统一桥梁。我们发现这些特征编码了从高层语义到细粒度结构的完整信息谱。为有效利用该特性，采用主子空间投影剔除导致“纹理烘焙”的高频成分，同时引入随机通道尾部丢弃以缓解刚性降维带来的结构损失，从而调和真实感与控制一致性。为进一步发挥DINOv3的高分辨率优势以增强控制精度，提出可学习空间对齐模块，适配高分辨率特征至扩散主干。最后，设计基于因果卷积的因果时序聚合器，显式保留帧间历史运动上下文，在整合分帧DINO特征时有效减轻运动模糊并确保时序稳定性。
原文摘要 · Abstract (English)
Driven by the emergence of Controllable Video Diffusion, existing Sim2Real methods for autonomous driving video generation typically rely on explicit intermediate representations to bridge the domain gap. However, these modalities face a fundamental Consistency-Realism Dilemma. Low-level signals (e.g., edges, blurred images) ensure precise control but compromise realism by "baking in" synthetic artifacts, whereas high-level priors (e.g., depth, semantics, HDMaps) facilitate photorealism but lack the structural detail required for consistent guidance. In this work, we present Driving with DINO (DwD), a novel framework that leverages Vision Foundation Module (VFM) features as a unified bridge between the simulation and real-world domains. We first identify that these features encode a spectrum of information, from high-level semantics to fine-grained structure. To effectively utilize this, we employ Principal Subspace Projection to discard the high-frequency elements responsible for "texture baking," while concurrently introducing Random Channel Tail Drop to mitigate the structural loss inherent in rigid dimensionality reduction, thereby reconciling realism with control consistency. Furthermore, to fully leverage DINOv3's high-resolution capabilities for enhancing control precision, we introduce a learnable Spatial Alignment Module that adapts these high-resolution features to the diffusion backbone. Finally, we propose a Causal Temporal Aggregator employing causal convolutions to explicitly preserve historical motion context when integrating frame-wise DINO features, which effectively mitigates motion blur and guarantees temporal stability. Project page: https://albertchen98.github.io/DwD-project/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。