让视觉语言模型像司机一样用工具推理，大幅减少错误判断。
AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving
- 用动态工具库自动生成带验证的结构化推理数据
- 在DriveLMM-o1上推理得分提升53.91%，准确率提高33.54%
- 适合研究自动驾驶智能体与多工具协同的开发者
视觉语言模型在自动驾驶中前景广阔，但存在幻觉、推理低效和真实场景验证不足等问题，影响感知精度与步骤推理鲁棒性。为此，我们提出首个统一框架AgentThink，将链式思维（CoT）与类智能体动态工具调用结合。核心创新包括：(i) 结构化数据生成，建立自动驾驶工具库，自动生成包含工具使用的结构化、自验证推理数据；(ii) 两阶段训练流程，采用监督微调（SFT）与组相对策略优化（GRPO），赋予视觉语言模型自主调用工具能力；(iii) 类智能体工具使用评估，引入新型多工具评测协议，严格检验模型工具调用与使用效果。在DriveLMM-o1基准测试中，AgentThink使整体推理得分提升53.91%，答案准确率提升33.54%，显著改善推理质量与一致性。消融实验及跨多个基准的零样本/少样本泛化测试进一步验证其强大性能。这些成果为构建可信、工具感知的自动驾驶模型指明了新方向。代码已开源：https://github.com/curryqka/AgentThink。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) show promise for autonomous driving, yet their struggle with hallucinations, inefficient reasoning, and limited real-world validation hinders accurate perception and robust step-by-step reasoning. To overcome this, we introduce \textbf{AgentThink}, a pioneering unified framework that integrates Chain-of-Thought (CoT) reasoning with dynamic, agent-style tool invocation for autonomous driving tasks. AgentThink's core innovations include: \textbf{(i) Structured Data Generation}, which establishes an autonomous driving tool library to automatically construct structured, self-verified reasoning data explicitly incorporating tool usage for diverse driving scenarios; \textbf{(ii) A Two-stage Training Pipeline}, employing Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO) to equip VLMs with the capability for autonomous tool invocation; and \textbf{(iii) Agent-style Tool-Usage Evaluation}, introducing a novel multi-tool assessment protocol to rigorously evaluate the model's tool invocation and utilization. Experiments on the DriveLMM-o1 benchmark demonstrate that AgentThink significantly boosts overall reasoning scores by \textbf{53.91%} and enhances answer accuracy by \textbf{33.54%}, while markedly improving reasoning quality and consistency. Furthermore, ablation studies and robust zero-shot/few-shot generalization experiments across various benchmarks underscore its powerful capabilities. These findings highlight a promising trajectory for developing trustworthy and tool-aware autonomous driving models. Code is available at https://github.com/curryqka/AgentThink.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。