arXiv:2601.15335cs.SEcs.AI2026-01被引 2

为大模型工具调用设计高效缓存系统,减少重复请求延迟。

ToolCaching: Towards Efficient Caching for LLM Tool-calling

  • 基于语义与系统特征动态评估请求是否适合缓存
  • 新算法使缓存命中率提升11%,延迟降低34%
  • 适合需要高频调用外部API的大模型应用开发者

大型语言模型(LLMs)的进展推动了智能搜索、推荐和助手服务的发展,自然语言接口使其更具实用性。工具调用扩展了模型与外部API交互的能力,显著提升了实际应用价值。尽管已有研究通过并行与异步执行等传统计算机技术优化工具调用性能,但冗余或重复请求的问题仍缺乏有效解决。缓存是经典方案,但在大模型工具调用中因请求语义异构、工作负载动态变化及新鲜度要求差异,传统缓存策略失效。为此,我们提出ToolCaching——一种面向大模型工具调用的高效特征驱动自适应缓存框架。该框架系统性融合语义与系统级特征,评估请求缓存可行性并估算缓存价值。核心的VAAC算法结合基于强化学习的准入机制与多因素价值驱动的淘汰策略,综合考虑请求频率、时效性与缓存收益。在合成与公开工具调用工作负载上的大量实验表明,采用VAAC的ToolCaching相比标准策略,缓存命中率最高提升11%,延迟降低34%,显著加速了实际应用中的大模型工具调用。

原文摘要 · Abstract (English)

Recent advances in Large Language Models (LLMs) have revolutionized web applications, enabling intelligent search, recommendation, and assistant services with natural language interfaces. Tool-calling extends LLMs with the ability to interact with external APIs, greatly enhancing their practical utility. While prior research has improved tool-calling performance by adopting traditional computer systems techniques, such as parallel and asynchronous execution, the challenge of redundant or repeated tool-calling requests remains largely unaddressed. Caching is a classic solution to this problem, but applying it to LLM tool-calling introduces new difficulties due to heterogeneous request semantics, dynamic workloads, and varying freshness requirements, which render conventional cache policies ineffective. To address these issues, we propose ToolCaching, an efficient feature-driven and adaptive caching framework for LLM tool-calling systems. ToolCaching systematically integrates semantic and system-level features to evaluate request cacheability and estimate caching value. At its core, the VAAC algorithm integrates bandit-based admission with value-driven, multi-factor eviction, jointly accounting for request frequency, recency, and caching value. Extensive experiments on synthetic and public tool-calling workloads demonstrate that ToolCaching with VAAC achieves up to 11% higher cache hit ratios and 34% lower latency compared to standard policies, effectively accelerating LLM tool-calling in practical applications.

大模型缓存优化工具调用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。