提出首个评估长时记忆型智能体主动询问能力的基准测试
Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
- 设计ATRBench基准，评估智能体是否在当前不需但未来可用时主动询问用户偏好
- 八款前沿智能体平均表现比理想情况低62分以上，提示也难弥补差距
- 揭示获取用户隐性偏好是当前智能体的核心短板，适合研究交互主动性的学者
长时运行的LLM智能体（如OpenClaw）的价值在于跨会话理解并响应用户偏好与约束，而非仅回应当前请求。然而，现有智能体多被动接收用户信息，极少主动追问未言明的需求，导致‘主动性缺口’——无法行动于从未获取的偏好。随着用户将更多事务委托给智能体，这一缺口影响日益显著。本文聚焦其中可控制的一环：‘现在问，以后用’（Ask-to-Remember, ATR），即智能体判断是否在当前任务无需、但未来可能需要时主动询问可复用的用户偏好。该问题难以评估，因提问时机不确定且收益延迟至可能永不发生的任务。为此，我们构建了ATRBench，据知是首个专门评估ATR的基准，通过设定用户偏好为隐藏真值，要求智能体真正‘询问’而非依赖回忆。在八款前沿智能体上，其默认表现至少比拥有理想偏好的奥拉克（oracle）低62分，提示工程也无法显著改善。诊断分析表明，信息获取是瓶颈。ATRBench揭示了当前智能体的主动性缺陷，并提供了一个可诊断、可改进的测试平台。
原文摘要 · Abstract (English)
A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents, the impact of this gap grows. We isolate one concrete, controllable slice of this gap as Ask-to-Remember (ATR): the agent decides whether to ask now for a reusable user preference that the current task does not need but a later session with the same user will. ATR is hard even to evaluate: the right question is underdetermined and its payoff deferred to tasks that may never arise. ATRBench, to the best of our knowledge the first ATR benchmark, makes it measurable by fixing each user's preferences as hidden ground truth, so success demands asking, not recall. Across eight frontier LLM agents, defaults fall at least 62 points below an oracle handed the relevant preference, and prompting closes little of it. Diagnostics identify acquisition as the bottleneck. ATRBench surfaces this proactivity gap in current agents and offers a diagnostic testbed for closing it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。