通过缓存任务计划模板，显著降低大模型代理的推理成本与延迟。
Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents
- 在测试时提取并复用相似任务的结构化计划模板。
- 相比原系统，平均节省50.31%成本，降低27.28%延迟。
- 适合需要高效执行复杂任务的大模型代理应用。
基于大模型的智能体应用在复杂工作流中展现出强大能力，但因大量规划与推理需求导致高昂成本与延迟。现有缓存技术（如上下文缓存、语义缓存）主要面向对话服务，难以应对依赖外部数据和环境上下文的智能体场景。本文提出一种新型测试时记忆机制——代理计划缓存（Agentic Plan Caching, APC），从已完成的智能体执行中提取、存储、适配并复用结构化的计划模板，以降低服务成本与延迟。该方法在测试时动态生成计划模板，利用关键词匹配新请求，并通过轻量模型将缓存模板适配至具体任务上下文。在多个真实世界智能体应用上的评估显示，APC可平均减少50.31%的成本与27.28%的延迟，同时保持性能，为大模型智能体服务提供了高效补充方案。
原文摘要 · Abstract (English)
LLM-based agent applications have shown increasingly remarkable capabilities in complex workflows but incur substantial costs and latency due to extensive planning and reasoning requirements. Existing LLM caching techniques (like context caching and semantic caching), primarily designed for serving chatbots, are insufficient for agent applications where outputs depend on external data and environmental contexts. We propose Agentic Plan Caching (APC), a novel test-time memory that extracts, stores, adapts, and reuses structured plan templates from planning stages of agent applications across semantically similar tasks to reduce the cost and latency of serving. Unlike traditional semantic caching, our system extracts plan templates from completed agent executions at test-time, employs keyword extraction to match new requests against cached plans, and utilizes lightweight models to adapt these templates to task-specific plans with contexts. Evaluation across multiple real-world agent applications shows that our system can reduce costs by 50.31% and latency by 27.28% on average while maintaining performance, offering a more efficient solution for serving LLM-based agents that complements existing LLM serving infrastructures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。