对比工具与规划对大模型推理的增益，发现效果受任务类型和成本限制显著影响。
When Do Tools and Planning Help Large Language Models Think? A Cost- and Latency-Aware Benchmark
- 设计带工具的规划代理，对比单次提示与多步执行
- 事件问答任务中准确率提升20个百分点，但延迟增加近40倍
- 小模型在复杂工具调度中表现下降，需权衡成本与性能
现代大语言模型越来越多地依赖推理时的规划和外部工具来提升推理能力。我们在两个真实场景中进行基准测试：基于图结构知识的事件中心问答（Event-QA）和Reddit ChangeMyView（CMV）中的说服性回复生成。使用LangChain和LangGraph，对比单次提示基线与配备任务专用工具（DBpedia SPARQL/查找/模式探索、以维基百科为中心的检索、主题网络搜索）的计划-执行-重规划代理。在Event-QA和CMV各取60个样本（每组3个20样本子集），报告平均端到端延迟和每个样本的标记成本估算。在相同工作流下评估GPT-4o和GPT-4o-mini的准确率与端到端延迟。在Event-QA任务中，最佳工具增强配置将准确率从47.5%提升至67.5%（以GPT-4o为例），但延迟从约8秒增至约317秒。在CMV任务中，单次提示表现最优（如GPT-4o-mini在约6秒内达到75%准确率），而规划+搜索显著增加延迟且未带来稳定提升。然而，复杂的多工具编排暴露了失败模式，小模型性能反而下降。总体表明，必须根据任务特性，做出兼顾成本与延迟的模型大小及智能体/工具复杂度选择。
原文摘要 · Abstract (English)
Modern large language models (LLMs) increasingly rely on inference-time planning and external tools to improve reasoning. We benchmark this behavior on two real-world settings: event-centric question answering over graph-structured knowledge (Event-QA) and persuasive response generation in Reddit ChangeMyView (CMV). Using LangChain and LangGraph, we compare a one-shot baseline against a plan-execute-replan agent equipped with task-specific tools (DBpedia SPARQL/lookup/schema exploration, Wikipedia-focused retrieval, and topical web search). We evaluate on 60 examples each from Event-QA and CMV (3 splits of 20), and report both mean end-to-end latency and per-example token cost estimates. We evaluate GPT-4o and GPT-4o-mini under identical workflows and report accuracy and end-to-end latency. On Event-QA, the best tool-augmented configuration improves accuracy (e.g., 47.5\% $\rightarrow$ 67.5\% for GPT-4o) while increasing latency by orders of magnitude ($\sim$8s $\rightarrow$ $\sim$317s per example). On CMV, one-shot prompting is strongest (e.g., GPT-4o-mini achieves 75\% at $\sim$6s), and planning+search increases latency substantially without consistent gains. However, complex multi-tool orchestration exposes failure modes where the smaller model degrades. Overall, the findings highlight the need for task-specific, cost-aware choices of both model size and agent/tooling complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。