统一多模态自动驾驶数据,让跨数据集研究变得简单。
123D: Unifying Multi-Modal Autonomous Driving Data at Scale

- 用统一API管理不同传感器数据流,支持任意时间同步访问
- 整合8个真实数据集共3300小时、9万公里,还包含可配置的合成数据
- 适合做跨数据集模型训练与评估的研究者使用
自动驾驶研究产生了机器人领域最丰富的传感器数据集合,但其规模与多样性仍远未被充分利用。各数据集采用不同的2D和3D模态(如摄像头、激光雷达、车辆状态、标注、交通灯、高精地图),采样率和同步方式各异,格式碎片化,难以在统一环境中共存。此外,标注规范不一致阻碍了跨数据集的模型训练与泛化评估。本文提出123D开源框架,通过单一API统一多模态驾驶数据。为处理同步问题,将每种模态作为独立的时间戳事件流存储,无固定采样率,支持跨数据集的同步或异步访问。利用123D,我们整合了8个真实世界驾驶数据集(总计3,300小时,90,000公里),并加入一个可配置采集脚本的合成数据集,提供数据分析与可视化工具。我们系统比较了标注统计,并评估各数据集的位姿与标定精度。进一步展示了两个应用:跨数据集3D目标检测迁移与强化学习规划,并提出未来方向建议。代码与文档见https://github.com/kesai-labs/py123d。
原文摘要 · Abstract (English)
The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each dataset adopts different 2D and 3D modalities, such as cameras, lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple datasets. We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, we store each modality as an independent timestamped event stream with no prescribed rate, enabling synchronous or asynchronous access across arbitrary datasets. Using 123D, we consolidate eight real-world driving datasets spanning 3,300 hours and 90,000 kilometers, together with a synthetic dataset with configurable collection scripts, and provide tools for data analysis and visualization. We conduct a systematic study comparing annotation statistics and assessing each dataset's pose and calibration accuracy. Further, we showcase two applications 123D enables: cross-dataset 3D object detection transfer and reinforcement learning for planning, and offer recommendations for future directions. Code and documentation are available at https://github.com/kesai-labs/py123d.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。