arXiv:2411.02623cs.AIcs.CY2024-11NeurIPS被引 16

不猜人类意图,直接提升其行动影响力来辅助决策

Learning to Assist Humans without Inferring Rewards

  • 用对比式后继表示法增强人类对环境的控制力
  • 在合成任务和《Overcooked》游戏中表现优于现有方法
  • 适合研究人机协作与可扩展辅助系统的学者

助人型智能体应让人类生活更轻松。传统方法通过逆强化学习推断人类意图,再选择帮助动作,但高维场景下意图推断困难。本文从赋能(empowerment)视角出发,让助人智能体最大化人类行动对环境结果的影响,从而以更少步骤完成任务。我们引入对比式后继表示法,克服了此前方法在高维场景中不可扩展的局限。理论证明该表示能准确估计赋能概念,并提供可优化机制。实验表明,该方法在合成基准上优于先前方法,且成功拓展至《Overcooked》这一协作游戏场景。本工作融合信息论、神经科学与强化学习思想,为表征在助人问题中的核心作用提供了新路径。

原文摘要 · Abstract (English)

Assistive agents should make humans' lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human's intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human's actions such that they exert a greater control over the environmental outcomes and can solve tasks in fewer steps. We lift the major limitation of prior work in this area--scalability to high-dimensional settings--with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a ready-made mechanism for optimizing it. Empirically, our proposed method outperforms prior methods on synthetic benchmarks, and scales to Overcooked, a cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts a path for representations to play a critical role in solving assistive problems.

人机协作赋能表示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。