通过双粒度管理降低大模型代理长时任务的上下文开销
TokenPilot: Cache-Efficient Context Management for LLM Agents
- 分层管理：全局压缩稳定提示前缀，局部淘汰保留任务相关性
- 连续模式下成本降低87%，性能优于已有系统
- 适合需要长时间运行的大模型应用开发人员
随着大模型代理在长时间会话中部署，上下文累积导致推理成本上升。现有方法通过文本修剪或动态内存淘汰来减少令牌占用，但无约束的序列变更会破坏布局，引发前缀不匹配和缓存失效。这暴露了文本稀疏性与提示缓存连续性之间的关键权衡。为此，我们提出TokenPilot，一种双粒度上下文管理框架。全局层面，摄入感知压缩在摄入阶段稳定提示前缀并消除开放环境噪声；局部层面，生命周期感知淘汰监控上下文片段的持续剩余价值，仅在任务相关性耗尽时保守地批量卸载内容。在PinchBench和Claw-Eval上的实验表明，无论是孤立模式还是连续模式，TokenPilot均实现61%与56%、以及61%与87%的成本降低，同时保持与先前系统的竞争力表现。TokenPilot已集成至LightRSI（https://github.com/zjunlp/RSI）。
原文摘要 · Abstract (English)
As LLM agents are deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimize token footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity and prompt cache continuity. To address this, we present TokenPilot, a dual-granularity context management framework. Globally, Ingestion-Aware Compaction acts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally, Lifecycle-Aware Eviction monitors the ongoing residual utility of context segments, enforcing a conservative batch-turn schedule to offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated and continuous modes demonstrate that TokenPilot reduces costs by 61% and 56% in isolated mode, and 61% and 87% in continuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightRSI at https://github.com/zjunlp/RSI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。