通过增强策略一致性,减少行为克隆集成中的动作分歧。
Swarm Behavior Cloning
- 用集成行为克隆提升多样性,同时优化策略间对齐度。
- 在8个环境中动作差异显著降低,平均回报明显提升。
- 适合需要稳定鲁棒决策的强化学习应用。
在序列决策环境中,主流训练方法为强化学习(RL)和模仿学习(IL)。与依赖奖励函数的RL不同,IL利用专家演示,即由专家策略 $π_e$(如人类)提供期望行为。形式上,给定状态-动作对数据集 $D = {(s, a = π_e(s))}$。模仿学习中常用的方法是行为克隆(BC),即通过监督学习从 $D$ 学习策略 $π(s) = a$。进一步改进可通过训练 $N$ 个独立的BC策略构成集成 $E = {π_i(s)}_{1 \leq i \leq N}$,其动作 $a$ 为各策略输出的均值:$a = \frac{1}{N} \sum_i π_i(s)$。本文解决了一个关键问题:在训练数据覆盖不足的状态下,各策略预测动作的差异会增大,导致聚合动作性能下降。为此,提出一种新方法,在保持计算多样性的同时增强策略间的对齐。该方法有效减小动作差异,同时保留集成的鲁棒性和多样决策优势。在8个不同环境中验证,动作差异显著下降,整体性能(以平均回合回报衡量)大幅提升。
原文摘要 · Abstract (English)
In sequential decision-making environments, the primary approaches for training agents are Reinforcement Learning (RL) and Imitation Learning (IL). Unlike RL, which relies on modeling a reward function, IL leverages expert demonstrations, where an expert policy $π_e$ (e.g., a human) provides the desired behavior. Formally, a dataset $D$ of state-action pairs is provided: $D = {(s, a = π_e(s))}$. A common technique within IL is Behavior Cloning (BC), where a policy $π(s) = a$ is learned through supervised learning on $D$. Further improvements can be achieved by using an ensemble of $N$ individually trained BC policies, denoted as $E = {π_i(s)}{1 \leq i \leq N}$. The ensemble's action $a$ for a given state $s$ is the aggregated output of the $N$ actions: $a = \frac{1}{N} \sum{i} π_i(s)$. This paper addresses the issue of increasing action differences -- the observation that discrepancies between the $N$ predicted actions grow in states that are underrepresented in the training data. Large action differences can result in suboptimal aggregated actions. To address this, we propose a method that fosters greater alignment among the policies while preserving the diversity of their computations. This approach reduces action differences and ensures that the ensemble retains its inherent strengths, such as robustness and varied decision-making. We evaluate our approach across eight diverse environments, demonstrating a notable decrease in action differences and significant improvements in overall performance, as measured by mean episode returns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。