通过推测机制优化大模型搜索代理，显著降低延迟。
Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design
- 提出两阶段自适应推测算法，安全时跳过验证步骤。
- 实测端到端速度提升最高达1.65倍，准确率不降反升。
- 适合追求低延迟的多步推理系统部署场景。
基于大模型的搜索代理虽性能优异，但因每一步需串行完成大模型推理和工具执行，导致严重延迟。本文从推测视角重新审视这一瓶颈。传统预测-验证模式虽能打破串行依赖，但受限于保留完整原始工作负载并引入额外推理开销。研究发现，早期代理步骤多为简单信息搜集，正确动作常可无需完整推理即预测得出。基于此，本文提出SPAgent——一种算法-系统协同设计框架，将推测作用扩展至搜索代理中以降低延迟。算法层面，引入两阶段自适应推测机制，在安全情况下选择性跳过验证；系统层面，采用两级调度器根据引擎负载动态调控推测请求，确保推测始终带来收益。在真实系统中实现SPAgent后，广泛实验表明其端到端速度最高提升1.65倍，且准确率保持不变甚至更高，推动多步搜索代理的实际应用。
原文摘要 · Abstract (English)
LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to $1.65\times$ end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。