让自动驾驶模型从第一视角图像中自博弈训练，突破感知鸿沟。
Pictura: Perspective-View Self-Play at Scale for Driving
- 构建第一视角自博弈训练框架，直接用摄像头输入训练驾驶模型。
- 在单张H100上实现50万智能体步/秒，训练超3500万公里里程。
- 零样本迁移至真实数据集布局，表现超越依赖精确感知的基准模型。
模拟中的自博弈可大规模生成鲁棒驾驶策略。已有研究利用精确位姿、速度等特权向量观测训练模型，即使对被遮挡目标也有效，但这假设感知已解决，与部署时前视摄像头的局部观测存在表征差距。常见做法是将特权策略蒸馏为仅接收相机输入的学生模型，但学生难以解释自身视角下的决策。为此，我们提出第一视角自博弈训练范式。引入Pictura——一个基于GPU加速的多智能体驾驶仿真器，每一步渲染每个代理的前视视角，从源头缓解表征差距。Pictura在单张H100上可持续达50万智能体步/秒（200万图像/秒）。基于此，我们使用普通PPO算法训练了Alberti，它是首个完全从第一视角图像出发、无需特权观测的大规模自博弈驾驶策略。训练涵盖500亿智能体步，约3500万公里行驶距离。其驾驶性能接近具备特权观测的对应模型，并能零样本迁移到在Pictura中重渲染的Waymo Open Motion Dataset布局，表现优于依赖特权观测的代理。
原文摘要 · Abstract (English)
Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-agent driving simulator that renders each agent's egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B agent steps for ~35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. Project page: https://valeoai.github.io/Pictura/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。