对比了五种LLM长时记忆系统在云边协同下的成本与精度表现。
Cost and Accuracy of Long-Term Memory in Distributed Multi-Agent Systems Based on Large Language Models
- 构建云边模拟环境，量化评估五类系统的性能指标。
- RAG和全上下文基线达77%-81%准确率，成本仅为mem0的1/8.4。
- 向量存储精度决定记忆质量，而非上下文长度，适合资源受限场景。
长时记忆（LTM）是基于大语言模型（LLM）的智能体在新兴物联网智能体（IoA）中的核心能力，分布式多智能体系统（DMAS）横跨云与边缘网络。现有评估多由框架厂商发布，侧重令牌使用与延迟，极少考虑系统级成本或在DMAS中的部署表现。本文构建独立可复现的测试平台，在模拟的云边环境中评估准确性、延迟、CPU时间、峰值内存、磁盘I/O及网络使用情况。对比了三种获得风投支持的架构（mem0、Graphiti、cognee）以及检索增强生成（RAG）和全上下文基线，在LoCoMo基准上进行无约束与受约束网络条件下的测试。结果形成两个聚类：mem0、RAG与全上下文达到77%至81%准确率，而Graphiti与cognee仅达55%至56%，差距源于检索不完整而非推理失败。RAG基线在总拥有成本（TCO）上比mem0低8.4倍，且两者均位于帕累托前沿非占优区域。延迟与带宽限制及抖动对所有后端的检索质量无影响，而向量型LTM在云边约束下仅带来4%至5%的延迟增加。压缩精度而非上下文体积决定LTM准确率，全上下文转发虽提供完整对话仍逊于mem0。
原文摘要 · Abstract (English)
Long-term memory (LTM) is fundamental to large language model (LLM)-based agents in the emerging Internet of Agents (IoA), where distributed multi-agent systems (DMAS) span cloud and edge networks. Existing evaluations are typically published by framework providers and focus on token usage and latency, rarely accounting for system-level cost or deployment in DMAS. These gaps are addressed with an independent reproducible testbed that evaluates accuracy, latency, CPU time, peak RAM, disk I/O and network usage in a simulated cloud-edge environment. Three venture capital-funded frameworks spanning vector, graph, and hybrid architectures, namely mem0, Graphiti, and cognee, are compared alongside retrieval-augmented generation (RAG) and full-context baselines on the LoCoMo benchmark under unconstrained and constrained network scenarios. Two clusters emerge: mem0, RAG, and full-context reach 77% to 81% accuracy, while Graphiti and cognee reach only 55% to 56%, a gap driven by retrieval incompleteness rather than reasoning failure. The RAG baseline matches the upper cluster at 8.4 times lower total cost of ownership (TCO) than mem0, and both are the only non-dominated backends on the Pareto frontier. Latency and bandwidth constraints as well as jitter leave retrieval quality unchanged for every backend, while vector-based LTM incurs a modest latency penalty of 4% to 5% under edge-cloud constraints. Compression precision rather than context volume determines LTM accuracy, as full-context forwarding underperforms mem0 despite supplying the entire conversation for each question.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。