提出可泛化的摄像头模拟框架,支持任意视角和帧率的可控生成。
DriveCamSim: Generalizable Camera Simulation via Explicit Camera Modeling for Autonomous Driving
- 通过显式像素对应建立多视角多帧关联,摆脱对固定参数的依赖。
- 在不同相机参数和帧率下均保持高质量图像与时间一致性。
- 适合自动驾驶视觉算法测试、数据增强等需要灵活模拟的场景。
摄像头传感器仿真在自动驾驶(AD)中至关重要,例如用于评估基于视觉的自动驾驶算法。现有方法虽借助生成模型实现可控图像/视频生成,但仅限于固定视角和固定帧率的多视图序列,严重限制了下游应用。为此,我们提出通用摄像头模拟框架DriveCamSim,核心创新为显式摄像头建模(ECM)机制。不同于传统的隐式注意力交互,ECM在多视角与多帧维度上建立显式的像素级对应关系,使模型摆脱对训练数据中特定相机配置(内参/外参、视角数量)和采样频率的过拟合。针对现有条件编码与注入管道中存在的信息丢失问题,我们提出信息保持的控制机制,不仅提升了条件可控性,还可扩展为身份感知设计,增强前景物体渲染的时间一致性。上述设计使模型在视觉质量、可控性及泛化能力方面表现优异,能有效应对空间层面(相机参数变化)和时间层面(帧率变化)的差异,支持用户自定义的灵活摄像头模拟,适配多样应用场景。代码将开源至 https://github.com/swc-17/DriveCamSim,以促进后续研究。
原文摘要 · Abstract (English)
Camera sensor simulation serves as a critical role for autonomous driving (AD), e.g. evaluating vision-based AD algorithms. While existing approaches have leveraged generative models for controllable image/video generation, they remain constrained to generating multi-view video sequences with fixed camera viewpoints and video frequency, significantly limiting their downstream applications. To address this, we present a generalizable camera simulation framework DriveCamSim, whose core innovation lies in the proposed Explicit Camera Modeling (ECM) mechanism. Instead of implicit interaction through vanilla attention, ECM establishes explicit pixel-wise correspondences across multi-view and multi-frame dimensions, decoupling the model from overfitting to the specific camera configurations (intrinsic/extrinsic parameters, number of views) and temporal sampling rates presented in the training data. For controllable generation, we identify the issue of information loss inherent in existing conditional encoding and injection pipelines, proposing an information-preserving control mechanism. This control mechanism not only improves conditional controllability, but also can be extended to be identity-aware to enhance temporal consistency in foreground object rendering. With above designs, our model demonstrates superior performance in both visual quality and controllability, as well as generalization capability across spatial-level (camera parameters variations) and temporal-level (video frame rate variations), enabling flexible user-customizable camera simulation tailored to diverse application scenarios. Code will be avaliable at https://github.com/swc-17/DriveCamSim for facilitating future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。