通过分层记忆与工具协同,让自动驾驶模型边开边学、越跑越准。
Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving

- 构建分层记忆:短期记忆跟踪实时场景,长期记忆存储可复用经验与工具策略。
- 推理时动态调用工具,离线时固化经验,实现闭环自我进化,推理得分80.03,准确率79.09%。
- 首次实现工具与记忆的协同闭环,适合自动驾驶系统研发与具身智能研究者。
视觉语言模型在自动驾驶中展现潜力,但仍存在幻觉、时空感知弱和泛化能力差的问题。现有方法通过思维链、检索增强生成或静态注入工具输出来提升推理,但模型无法主动感知环境,也无法在回答后积累经验。为此,我们提出首个将分层记忆与主动工具调用紧密结合的协同推理框架。第一,构建分层驾驶记忆:场景级短期记忆维护动态环境状态,演化型长期记忆检索可复用的经验与工具策略。第二,记忆-工具协同推理框架:基于当前场景状态与检索经验,模型在推理时自适应调用工具优化判断,并离线将可复用经验存入长期记忆池。第三,数据生成与两阶段训练流程:通过多步教师演算构建验证后的记忆-工具轨迹,用于监督微调(SFT)和强化学习(GRPO)。7B模型在DriveLMM-o1上取得80.03的总体推理分和79.09%的MCQ准确率,超越最强基线7.74点。消融实验表明,短期记忆使时空理解提升24.2分,长期记忆离线固化带来3.57分增益(全参数冻结),证明持续自我进化能力。
原文摘要 · Abstract (English)
VLMs have shown promise for autonomous driving, yet still suffer from hallucination, weak spatio-temporal perception, and limited generalization. Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs. Although these mechanisms enrich the context, the model neither proactively perceives scene information nor accumulates experience after answering. To overcome these limitations, we present, to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed reasoning loop. Our contributions are threefold. (i) Hierarchical Driving Memory: a scene-level short-term memory maintains the dynamic scene state, and an evolving long-term memory retrieves reusable experience and tool strategies. (ii) Memory-Tool Synergistic Reasoning Framework: guided by the scene state and retrieved experience, the model adaptively invokes tools to refine its reasoning at inference time and consolidates reusable experience into a long-term memory pool offline. (iii) Data Generation and Two-stage Training Pipeline: verified memory-tool trajectories built by multi-step teacher rollout are used to train with SFT and GRPO. Our 7B model reaches an overall reasoning score of 80.03 and MCQ accuracy of 79.09% on DriveLMM-o1, surpassing the strongest baseline by 7.74 MCQ points and generalizes strongly across benchmarks. Notably, ablation and analysis studies validate the effectiveness of each component and further reveal the complementary roles of hierarchical memory. Short-term memory strengthens spatio-temporal understanding, improving STSBench accuracy by 24.2 points, while offline long-term memory consolidation yields an additional 3.57-point MCQ gain with all parameters frozen, demonstrating continual self-evolution through accumulated driving experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。