让强化学习更聪明地用示范数据，不盲目跟从。
APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Composition
- 分层模型自适应组合多个示范先验，动态判断是否可用
- 在示范不完美时仍能加速学习，避免性能下降
- 适合有示范但质量参差的复杂任务场景
将示范数据融入强化学习可显著加速学习过程，但现有方法通常假设示范数据是最优且与目标任务完全对齐的。实际中，示范数据常稀疏、次优或存在偏差，直接融合会降低性能。本文提出自适应策略组合（APC），一种分层模型，可自适应地组合多个基于归一化流（Normalizing Flow）的数据驱动先验。APC不强制遵循先验，而是估计每个先验对目标任务的适用性，并利用它们进行探索。当先验有用时进行优化，当存在偏差时则绕过，以最大化下游奖励。在多种基准测试中，当示范数据对齐时，APC能加速学习；在严重错位情况下仍保持鲁棒性；并能利用次优示范启动探索，同时避免因过度依赖次优示范导致的性能下降。
原文摘要 · Abstract (English)
Incorporating demonstration data into reinforcement learning (RL) can greatly accelerate learning, but existing approaches often assume demonstrations are optimal and fully aligned with the target task. In practice, demonstrations are frequently sparse, suboptimal, or misaligned, which can degrade performance when these demonstrations are integrated into RL. We propose Adaptive Policy Composition (APC), a hierarchical model that adaptively composes multiple data-driven Normalizing Flow (NF) priors. Instead of enforcing strict adherence to the priors, APC estimates each prior's applicability to the target task while leveraging them for exploration. Moreover, APC either refines useful priors, or sidesteps misaligned ones when necessary to optimize downstream reward. Across diverse benchmarks, APC accelerates learning when demonstrations are aligned, remains robust under severe misalignment, and leverages suboptimal demonstrations to bootstrap exploration while avoiding performance degradation caused by overly strict adherence to suboptimal demonstrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。