通过动态切换策略提升离线强化学习性能,兼顾数据安全与效果优化。
Evaluation-Time Policy Switching for Offline Reinforcement Learning
- 基于认知不确定性与数据异变性动态切换策略
- 在多个基准上超越单个算法及当前最优方法
- 可无缝迁移至离线到在线微调,无需调参
离线强化学习(Offline RL)旨在利用固定环境交互数据集学习最优策略。现有基于离线的算法虽能适配特定数据集,但面对不同任务或数据质量时需手动调参。本文提出一种策略切换技术,动态融合纯离线强化学习代理(用于提升行为表现)与行为克隆(BC)代理(用于贴近原始数据分布)。该方法结合模型自身的认知不确定性与从数据集中提取的随机不确定性度量实现决策。实验证明,该策略切换机制不仅优于参与切换的单一算法,还在多个基准测试中媲美甚至超越当前最先进方法。此外,利用认知不确定性进行策略切换,使本方法天然支持从离线到在线的快速安全微调,无需额外修改或超参数调整即可达到或超过现有方法性能。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) looks at learning how to optimally solve tasks using a fixed dataset of interactions from the environment. Many off-policy algorithms developed for online learning struggle in the offline setting as they tend to over-estimate the behaviour of out of distributions actions. Existing offline RL algorithms adapt off-policy algorithms, employing techniques such as constraining the policy or modifying the value function to achieve good performance on individual datasets but struggle to adapt to different tasks or datasets of different qualities without tuning hyper-parameters. We introduce a policy switching technique that dynamically combines the behaviour of a pure off-policy RL agent, for improving behaviour, and a behavioural cloning (BC) agent, for staying close to the data. We achieve this by using a combination of epistemic uncertainty, quantified by our RL model, and a metric for aleatoric uncertainty extracted from the dataset. We show empirically that our policy switching technique can outperform not only the individual algorithms used in the switching process but also compete with state-of-the-art methods on numerous benchmarks. Our use of epistemic uncertainty for policy switching also allows us to naturally extend our method to the domain of offline to online fine-tuning allowing our model to adapt quickly and safely from online data, either matching or exceeding the performance of current methods that typically require additional modification or hyper-parameter fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。