arXiv:2512.06721cs.AIcs.CL2025-12被引 3

让大模型在真实生活中主动提供帮助,只在需要时调用感官信息。

ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild

  • 按需分层感知:低功耗持续监听,关键时调用更丰富信息
  • 预测准确率高27.7%,误报率降低20.5%优于现有方法
  • 适合希望实现真实场景智能助手的开发者与研究者

近期研究开始探索通过自动利用上下文信息(如代码编辑、应用内建议)提供无感协助的主动型大语言模型(LLM)代理。然而,多数工作聚焦于短时任务或屏幕内上下文,而非持续感知并辅助用户全天候生活。实现此类真实环境中的协助需要持续感知用户周围环境,但可能带来显著系统开销。本文提出 ProAgent,一个端到端的主动代理系统,通过按需调用感官上下文实现真实环境中的协助。ProAgent首先采用按需分层感知,融合低成本上下文线索与按需触发的高阶感知,持续监测用户环境;再通过面向主动性的上下文提取,整合感官信息与用户偏好,构建分层上下文。随后,基于上下文的主动推理器推断用户需求,并调用外部工具提供主动协助。我们在增强现实眼镜上实现了 ProAgent,评估了其在公开数据集和真实世界数据集上的表现。结果表明,ProAgent 在主动预测准确率上比最先进基线最高提升 27.7%,误检率降低 20.5%。20 名用户的实验显示,85% 的参与者对 ProAgent 满意,并愿意在日常生活中使用。

原文摘要 · Abstract (English)

Recent studies have begun to explore proactive large language model (LLM) agents that provide unobtrusive assistance by automatically leveraging contextual information, such as in code editing and in-app suggestions. However, most focus on short, task-specific episodes or on-screen contexts, rather than continuously perceiving and assisting users throughout daily life. Enabling such in-the-wild assistance requires continuous sensing of users' surroundings, which can incur substantial system overhead. In this work, we propose ProAgent, an end-to-end proactive agent system that harnesses on-demand sensory contexts to provide in-the-wild assistance. ProAgent first employs on-demand tiered perception to continuously sense users' surroundings by integrating low-cost contextual cues with richer perception on demand, and uses proactive-oriented context extraction to derive hierarchical contexts integrating both sensory contexts and human preferences. ProAgent then employs a context-aware proactive reasoner to infer user needs and invokes external tools to deliver proactive assistance. We implement ProAgent on AR glasses and evaluate it on a public dataset and a real-world dataset. Results demonstrate that ProAgent achieves up to 27.7% higher proactive prediction accuracy and 20.5% lower false detection than state-of-the-art baselines. A user study with 20 participants shows that 85% were satisfied with ProAgent and willing to use it in daily life.

主动代理增强现实上下文感知大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。