评测智能助手在长期任务中主动发现用户隐含需求的能力
$π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
- 构建100个跨领域多轮任务，模拟用户逐步暴露需求的真实场景
- 发现现有助手在主动识别需求上仍困难，且任务完成与主动性的表现差异明显
- 强调历史交互对后续主动推理的关键作用，适合评估长期陪伴型AI
个人助理智能体（如OpenClaw）的兴起表明大语言模型在日常生活和工作中具有巨大潜力。核心挑战在于主动服务：用户常提出不完整请求，关键需求、约束或偏好往往未被明确表达。然而，现有基准很少评估智能体能否在用户明确说明前就识别并响应这些隐藏意图，尤其是在持续多轮交互中用户需求逐步显现的场景。为此，我们提出π-Bench，一个涵盖5个领域用户角色的100个多轮任务基准，通过引入隐藏用户意图、任务间依赖关系及跨会话连续性，评估智能体在长期交互中预见并满足用户需求的能力，同时衡量主动性与任务完成度。实验表明：（1）主动服务仍具挑战性；（2）任务完成与主动性存在显著差异；（3）前期交互经验对后期主动意图理解至关重要。
原文摘要 · Abstract (English)
The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $π$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $π$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。