让AI默默听人类游戏对话，自动提供辅助建议。
First Steps Towards Overhearing LLM Agents: A Case Study With Dungeons & Dragons Gameplay
- 用多模态音频-语言模型监听玩家对话，被动获取信息
- 人类评估显示部分模型能通过语音线索完成辅助任务
- 适合对沉浸式交互、非侵入式AI辅助感兴趣的开发者
现有研究多聚焦于主动对话的LLM代理。本文提出一种新范式——“旁听代理”，它们不主动发言，而是监听人类间的对话，在后台执行任务或提供建议。我们以《龙与地下城》游戏为案例，使用大型多模态音频-语言模型作为旁听代理，辅助游戏主持人（Dungeon Master）。通过人工评估验证其有效性，发现部分大模型具备利用隐含音频线索完成旁听任务的涌现能力。项目代码与Python库已开源，支持该范式进一步研究。
原文摘要 · Abstract (English)
Much work has been done on conversational LLM agents which directly assist human users with tasks. We present an alternative paradigm for interacting with LLM agents, which we call "overhearing agents". These overhearing agents do not actively participate in conversation -- instead, they "listen in" on human-to-human conversations and perform background tasks or provide suggestions to assist the user. In this work, we explore the overhearing agents paradigm through the lens of Dungeons & Dragons gameplay. We present an in-depth study using large multimodal audio-language models as overhearing agents to assist a Dungeon Master. We perform a human evaluation to examine the helpfulness of such agents and find that some large audio-language models have the emergent ability to perform overhearing agent tasks using implicit audio cues. Finally, we release Python libraries and our project code to support further research into the overhearing agents paradigm at https://github.com/zhudotexe/overhearing_agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。