让驾驶模型慢思快动,每50毫秒输出一次新指令,大幅提高自动驾驶成功率。
Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving

- 慢系统用70亿参数视觉语言模型低频理解场景,快系统每帧即时生成控制指令。
- 在CARLA上完成率从37.0提升至94.0,红灯违规减少三分之一。
- 零样本迁移至未见城镇,适合追求高实时性与强泛化的自动驾驶研究者。
大语言模型为端到端自动驾驶带来指令理解与场景推理能力,但其推理延迟与车辆控制频率冲突。现有闭环代理通过交替仿真周期调用模型、中间重复旧命令来掩盖差距,导致一半控制输出无视最新观测。本文提出快-慢架构:冻结的70亿参数视觉语言主干作为慢系统,以低频处理导航指令与视觉历史,将其各层键值缓存作为场景的持续表征;轻量级动作专家作为快系统,每仿真周期关注该缓存与当前图像,单次前向传播回归航点。由于缓存部署时滞后于真实世界,训练时引入随机过时,使训练与异步执行对齐。在CARLA的LangAuto-Short路线中,系统实现每50毫秒一次新控制,路线完成率从37.0升至94.0(对比帧跳过基线)。相同专家的帧跳过消融实验分离出两因素:专家自身提升驾驶得分,而每帧更新使完成率从82.1增至94.0,红灯违规减少三分之一。仅在一座城镇训练,专家零样本迁移到两座未见城镇,完成率达84-94%(基线为31-41%)。相比主干自身动作头,开放环路航点误差降低近四倍,单帧模型开销32毫秒,且不随历史长度增长,可在单张消费级显卡运行。
原文摘要 · Abstract (English)
Large language models bring instruction following and scene reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop agents hide this gap by invoking the model on alternate simulation ticks and replaying the previous command in between, so half of all control outputs ignore the newest observations. We present a fast-slow architecture that removes this compromise. A frozen 7B vision-language backbone acts as the slow system, digesting navigation instructions and visual history at low frequency while exposing its per-layer key-value cache as a standing representation of the scene. A lightweight action expert acts as the fast system, attending to this cache and to the current camera frame at every simulation tick to regress waypoints in a single forward pass. Since the cache lags behind the world at deployment, we train the expert under randomized staleness, aligning training with asynchronous execution. On LangAuto-Short routes in CARLA, our system produces fresh control at every 50 ms simulation tick and lifts route completion from 37.0 to 94.0 over the frame-skipping baseline. A frame-skip ablation with the same expert separates the two factors at work: the expert raises the driving score on its own, while per-tick freshness raises completion from 82.1 to 94.0 and cuts red-light violations by a third. Trained on a single town, the expert transfers zero-shot to two unseen towns, holding 84-94% route completion where the baseline reaches 31-41%. It reduces open-loop waypoint error by nearly a factor of four compared to the backbone's own action head, at a per-tick model cost of 32 ms that is independent of history length on a single consumer GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。