构建首个大规模个人记忆助手基准，测试设备端多轮对话记忆能力
MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale
- 用仿真代理生成50人15天的连贯对话数据，模拟真实个人记忆场景
- 发现记忆后端选择对内容准确率影响远超模型规模，Memobase到MemSearch提升32.5个百分点
- 权限控制失效：理想检索会泄露隐私，其他方案又过度保守；小模型搜索延迟仅增加几十毫秒
边缘部署的个人记忆助手需在本地处理私密人际对话，使用开放权重模型。现有记忆基准常忽视活动密集交互、第一人称视角及多会话连贯性。MemArena通过其MASim代理模拟器构建单一世界对话基准，涵盖50个代理持续15天（1030万对话文本标记，每代理每日24.1千条纯文本观察标记）。基于交互历史，协同生成六维评估的真实标签（回忆、推理、可信度等）。评估五种开放权重阅读器搭配不同记忆后端（Vanilla上下文、BM25-RAG、Oracle检索、Memobase、MemSearch）。关键发现：(1) 记忆后端选择显著影响内容准确性：在Qwen3-0.6B上，从Memobase切换至MemSearch提升32.5/19.2百分点，超过MemSearch模型扩展带来的10.6/6.8百分点增益；(2) 权限感知访问普遍失败：Oracle存在严重泄露，其他后端又过于保守不披露信息；(3) 搜索延迟仅在极小模型时显著：在Spark GB10边缘节点上，记忆搜索增加87/7/48毫秒（BM25-RAG/Memobase/MemSearch），占总时间到首次响应时间（TTFT）比例较小。代码、MASim模拟器及MemArena-L基准将在论文接受后公开。
原文摘要 · Abstract (English)
Edge-deployed personal memory assistants must handle private interpersonal conversations on-device with open-weight models. Yet, existing memory benchmarks often under-test the combination of activity-dense interaction, ego-centric perspective, and coherent multi-session worlds. MemArena fills these gaps with a single-world conversational benchmark built with its MASim agent simulator, for 50 agents over 15 days (10.3M dialog-text tokens, 24.1K text-only ego-observed tokens/agent/day). With the interaction history, it co-generates ground truth over six recall, reasoning, and trustworthiness evaluation dimensions. We evaluate five open-weight readers with Vanilla context, BM25-RAG, Oracle retrieval, Memobase, and MemSearch as memory backends. Three results stand out: (1) Memory-backend choice matters more for content accuracy: At Qwen3-0.6B, Memobase-to-MemSearch gains +32.5/+19.2 pp, exceeding MemSearch reader scaling (+10.6/+6.8 pp). (2) Permission-aware access fails universally, with Oracle leaking heavily and other backends too timid to disclose. (3) Search latency bites only at very small reader: on a Spark GB10 edge node, memory-search adds a moderate and fixed 87/7/48 ms (BM25-RAG/Memobase/MemSearch) that composes a small part of TTFT for most reader-backend combinations. Code, the MASim simulator, and the MemArena-L benchmark will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。