让智能体自动学会空间关系概念,提升强化学习的可解释性与适应性。
GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learning

- 用大语言模型提供初始概念表示,通过环境交互自主优化
- 在3个Atari游戏上性能媲美甚至超越人工设计概念的模型
- 适合需要可解释策略的复杂任务场景,如机器人控制
神经符号强化学习(NeSy-RL)结合符号推理与梯度优化,实现可解释且泛化能力强的策略。关系概念如“在...左边”或“靠近”是智能体感知与行动的基础构件。然而,传统方法需人工定义这些概念,限制了适应性,因概念语义随环境而异。本文提出GRAIL(通过交互学习实现关系概念具象化),一种通过环境互动自主具象化关系概念的框架。GRAIL利用大语言模型(LLMs)提供通用概念表征作为弱监督信号,再将其细化以捕捉特定环境语义。该方法同时缓解稀疏奖励信号和概念错位问题。在Kangaroo、Seaquest和Skiing三个Atari游戏上的实验表明,GRAIL在简化环境下性能与人工设计概念相当甚至更优,在完整环境中揭示了奖励最大化与高层目标达成之间的有意义权衡。
原文摘要 · Abstract (English)
Neuro-symbolic Reinforcement Learning (NeSy-RL) combines symbolic reasoning with gradient-based optimization to achieve interpretable and generalizable policies. Relational concepts, such as "left of" or "close by", serve as foundational building blocks that structure how agents perceive and act. However, conventional approaches require human experts to manually define these concepts, limiting adaptability since concept semantics vary across environments. We propose GRAIL (Grounding Relational Agents through Interactive Learning), a framework that autonomously grounds relational concepts through environmental interaction. GRAIL leverages large language models (LLMs) to provide generic concept representations as weak supervision, then refines them to capture environment-specific semantics. This approach addresses both sparse reward signals and concept misalignment prevalent in underdetermined environments. Experiments on the Atari games Kangaroo, Seaquest, and Skiing demonstrate that GRAIL matches or outperforms agents with manually crafted concepts in simplified settings, and reveals informative trade-offs between reward maximization and high-level goal completion in the full environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。