优化大模型代理任务的提示缓存策略,显著降本提速
Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
- 对比三种缓存策略,发现动态内容放末尾更高效
- 缓存使API成本降低41%-80%,首字延迟减少13%-31%
- 适合需长期交互的自动化系统开发者参考
大语言模型代理任务涉及大量工具调用和长上下文,虽主流厂商提供提示缓存以降低开销,但其在复杂多轮任务中的效果尚无系统研究。本文对OpenAI、Anthropic和Google三家平台进行评估,比较全上下文缓存、仅系统提示缓存及排除动态工具结果的缓存策略。实验基于DeepResearch Bench基准,覆盖500+代理会话,系统提示达10,000 tokens,执行真实网络搜索任务。结果表明,合理设计缓存块可使API成本下降41%-80%,首字时间缩短13%-31%。通过消融实验验证,在500-50,000 token提示长度与3-50次工具调用范围内,成本与延迟均呈线性改善(超过提供商最小缓存阈值后)。不同厂商策略差异明显,建议生产级系统采用动态内容置后、避免函数调用缓存等策略。
原文摘要 · Abstract (English)
Recent advancements in Large Language Model (LLM) agents have enabled complex multi-turn agentic tasks requiring extensive tool calling, where conversations can span dozens of API calls with increasingly large context windows. However, although major LLM providers offer prompt caching to reduce cost and latency, its benefits for agentic workloads remain underexplored in the research literature. To our knowledge, no prior work quantifies these cost savings or compares caching strategies for multi-turn agentic tasks. We present a comprehensive evaluation of prompt caching across three major LLM providers (OpenAI, Anthropic, and Google) and compare three caching strategies, including full context caching, system prompt only caching, and caching that excludes dynamic tool results. We evaluate on DeepResearch Bench, a multi-turn agentic benchmark where agents autonomously execute real-world web search tool calls to answer complex research questions, measuring both API cost and time to first token (TTFT) across over 500 agent sessions with 10,000-token system prompts. Our results demonstrate that prompt caching reduces API costs by 41-80% and improves time to first token by 13-31% across providers. We find that strategic prompt cache block control, such as placing dynamic content at the end of the system prompt, avoiding dynamic traditional function calling, and excluding dynamic tool results, provides more consistent benefits than naive full-context caching, which can paradoxically increase latency. An ablation study across prompt sizes (500-50,000 tokens) and tool call counts (3-50) demonstrates universal linear cost and TTFT benefits, after the provider caching token minimum, and reveal provider-specific strategy discrepancies across variants. We provide nuanced discussion and guidance for implementing prompt caching in production agentic systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。