通过动态评估记忆价值，控制长时运行大模型的延迟波动。
AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems
- 按记忆项价值动态分级管理，限制检索候选集大小
- 相比TTL机制，中位延迟降低4.2倍，99%分位延迟降4.4倍
- 适合对响应稳定性要求高的长期运行大模型系统
长期运行的大模型智能体需要持久化记忆来保持状态，但现有系统多采用基于时间的保留策略（如TTL）。尽管TTL能限制记忆项生命周期，却无法控制请求路径上的计算开销：随着保留项积累，检索候选集和向量相似度扫描规模会不可预测地增长，导致重尾延迟和吞吐不稳定。本文提出AMV-L（自适应记忆生命周期管理）框架，将智能体记忆视为可管理的系统资源。AMV-L为每条记忆项分配持续更新的价值评分，通过价值驱动的晋升、降级与淘汰机制维护生命周期层级；检索仅限于有界且层级感知的候选集，实现请求路径工作集与总保留内存解耦。我们在全栈大模型服务系统中实现AMV-L，与两种基线（TTL和LRU工作集策略）在相同长时运行负载下对比，均设定了固定的提示注入上限。相比TTL，AMV-L提升吞吐3.1倍，中位延迟降低4.2倍，p95延迟降低4.7倍，p99延迟降低4.4倍，超过2秒的请求占比从13.8%降至0.007%。相比LRU，AMV-L在中位/95%延迟略有上升（+26%/+3%）的前提下，显著改善极端尾部延迟（p99下降15%，>2秒请求减少98%），并降低约6%的每请求令牌开销，同时保持检索质量（价值均值相差约0-2%）。性能提升主要源于对检索集规模和向量搜索工作的约束，而非缩短提示长度。结果表明，长时运行大模型智能体的可预测性能依赖于对记忆工作集大小的显式控制及价值驱动的生命周期管理，而不仅是保留时间。
原文摘要 · Abstract (English)
Long-running LLM agents require persistent memory to preserve state across interactions, yet most deployed systems manage memory with age-based retention (e.g., TTL). While TTL bounds item lifetime, it does not bound the computational footprint of memory on the request path: as retained items accumulate, retrieval candidate sets and vector similarity scans can grow unpredictably, yielding heavy-tailed latency and unstable throughput. We present AMV-L (Adaptive Memory Value Lifecycle), a memory-management framework that treats agent memory as a managed systems resource. AMV-L assigns each memory item a continuously updated utility score and uses value-driven promotion, demotion, and eviction to maintain lifecycle tiers; retrieval is restricted to a bounded, tier-aware candidate set that decouples the request-path working set from total retained memory. We implement AMV-L in a full-stack LLM serving system and evaluate it under identical long-running workloads against two baselines: TTL and an LRU working-set policy, with fixed prompt-injection caps. Relative to TTL, AMV-L improves throughput by 3.1x and reduces latency by 4.2x (median), 4.7x (p95), and 4.4x (p99), while reducing the fraction of requests exceeding 2s from 13.8% to 0.007%. Compared to LRU, AMV-L trades a small regression in median/p95 latency (+26% / +3%) for improved extreme-tail behavior (-15% p99; -98% >2s) and lower token overhead (approximately 6% fewer tokens/request), while matching retrieval quality (value means within approximately 0-2%). The gains arise primarily from bounding retrieval-set size and vector-search work, not from shortening prompts. Our results show that predictable performance for long-running LLM agents requires explicit control of memory working-set size and value-driven lifecycle management, rather than retention time alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。