通过状态感知缓存减少大模型代理训练中工具调用耗时
TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents
- 构建带状态的工具调用缓存，按完整历史匹配查找
- 最高缓存命中率达70%，工具调用时间减少6.9倍
- 适用于需频繁调用外部工具的强化学习训练场景
在大语言模型代理的强化学习后训练中，外部工具调用耗时数秒甚至数分钟，导致GPU闲置，显著增加训练时间和成本。尽管并行回溯中存在大量重复工具调用，但直接缓存输出不正确，因为工具结果依赖于先前交互所引发的环境状态。我们提出TVCACHE，一种面向大模型代理后训练的状态化工具值缓存机制。TVCACHE维护已观测工具调用序列的树结构，通过最长前缀匹配进行缓存查找：仅当代理完整工具历史与先前执行序列完全一致时才命中，确保环境状态相同。在三类不同工作负载（终端任务、SQL生成、视频理解）上，TVCACHE实现最高70%的缓存命中率，工具调用执行时间中位数降低6.9倍，且无后训练奖励累积下降。
原文摘要 · Abstract (English)
In RL post-training of LLM agents, calls to external tools take several seconds or even minutes, leaving allocated GPUs idle and inflating post-training time and cost. While many tool invocations repeat across parallel rollouts and could in principle be cached, naively caching their outputs for reuse is incorrect since tool outputs depend on the environment state induced by prior agent interactions. We present TVCACHE, a stateful tool-value cache for LLM agent post-training. TVCACHE maintains a tree of observed tool-call sequences and performs longest-prefix matching for cache lookups: a hit occurs only when the agent's full tool history matches a previously executed sequence, guaranteeing identical environment state. On three diverse workloads-terminal-based tasks, SQL generation, and video understanding. TVCACHE achieves cache hit rates of up to 70% and reduces median tool call execution time by up to 6.9X, with no degradation in post-training reward accumulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。