通过分层动作量化让强化学习智能体更像人类
Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization
- 用两级向量量化将人类示范转为宏观动作
- 在D4RL上人类相似度更高，成功率相当或更好
- 适配多种算法，提升智能体可解释性
类人智能体是人工智能的长期目标。尽管表现强劲，大多数强化学习（RL）智能体仍仅受奖励驱动，行为常与人类不同，限制了可解释性和可靠性。本文提出一种新型类人强化学习框架，通过预测与人类行为高度一致的动作序列来最大化奖励。核心方法是分层宏观动作量化（HiMAQ），包含两级向量量化：底层将输入动作映射到细粒度子动作簇，高层将子动作簇聚合为动作簇。在D4RL基准上的大量评估显示，该分层方法优于非分层基线（MAQ），在保持与以往RL智能体相当或更优成功率的同时，显著提升人类相似度得分。改进效果在集成多种算法（IQL、SAC、RLPD）时均具泛化性。
原文摘要 · Abstract (English)
Human-like agents are a long-standing goal of artificial intelligence. Despite strong performance, most reinforcement learning (RL) agents remain reward-driven and often exhibit behaviors that differ from humans, limiting interpretability and reliability. In this work, we introduce a novel human-like RL framework that predicts action sequences closely aligned with human behaviors while maximizing rewards. Specifically, we encode human demonstrations into macro actions using a hierarchical macro action quantization approach (termed HiMAQ) consisting of two successive levels of vector quantization. The lower quantization level maps input actions to fine-grained subaction clusters, while the higher quantization level aggregates these subaction clusters into action clusters. Extensive evaluations on the D4RL benchmarks show that our hierarchical approach outperforms the non-hierarchical baseline (MAQ), achieving better human-likeness scores while maintaining comparable or better success rates than previous RL agents. The improvements generalize across integrations with various RL algorithms, namely IQL, SAC, and RLPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。