用少量非最优数据快速提炼出高效策略，提升离线强化学习效率
Offline Behavior Distillation
- 从次优数据中蒸馏专家行为，降低训练数据依赖
- 新目标函数使收敛速度更快，且在多种架构下表现稳定
- 理论证明更优的折扣复杂度，适合追求高效训练的研究者
大规模强化学习数据常被离线收集以训练策略，但数据量过大导致训练效率低下。为此，本文提出离线行为蒸馏（OBD），旨在从次优强化学习数据中合成有限的专家行为数据，实现快速策略学习。设计了两种基础目标函数DBC和PBC，通过比较蒸馏后策略与离线数据或近似专家策略的决策差异来评估蒸馏效果。由于双层优化难以求解，传统目标难以达到小值，PBC受其二次折扣复杂度$/mathcal{O}(1/(1-γ)^2)$影响性能。本文理论证明策略性能与动作价值加权决策差异等价，并提出动作价值加权PBC（Av-PBC）作为更优目标。优化该加权差异可获得线性折扣复杂度$/mathcal{O}(1/(1-γ))$的优越蒸馏保证。在多个D4RL数据集上的实验表明，Av-PBC显著提升蒸馏性能，加速收敛，并具备跨架构与优化器的强泛化能力。
原文摘要 · Abstract (English)
Massive reinforcement learning (RL) data are typically collected to train policies offline without the need for interactions, but the large data volume can cause training inefficiencies. To tackle this issue, we formulate offline behavior distillation (OBD), which synthesizes limited expert behavioral data from sub-optimal RL data, enabling rapid policy learning. We propose two naive OBD objectives, DBC and PBC, which measure distillation performance via the decision difference between policies trained on distilled data and either offline data or a near-expert policy. Due to intractable bi-level optimization, the OBD objective is difficult to minimize to small values, which deteriorates PBC by its distillation performance guarantee with quadratic discount complexity $\mathcal{O}(1/(1-γ)^2)$. We theoretically establish the equivalence between the policy performance and action-value weighted decision difference, and introduce action-value weighted PBC (Av-PBC) as a more effective OBD objective. By optimizing the weighted decision difference, Av-PBC achieves a superior distillation guarantee with linear discount complexity $\mathcal{O}(1/(1-γ))$. Extensive experiments on multiple D4RL datasets reveal that Av-PBC offers significant improvements in OBD performance, fast distillation convergence speed, and robust cross-architecture/optimizer generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。