用少量干净数据识别并过滤脏数据,提升强化学习的鲁棒性。
Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
- 通过判别器估计轨迹密度比,加权重要性筛选样本
- 在80%污染数据下仍保持接近最优性能
- 无需了解污染机制,适合安全关键场景
离线强化学习(Offline RL)可在固定数据集上优化策略,适用于无法在线探索的安全关键场景。然而,这些数据集常受对抗性污染、系统错误或低质量样本影响,导致传统行为克隆(BC)和离线RL方法性能下降。本文提出密度比加权行为克隆(Weighted BC),利用少量已验证的干净参考集,通过二元判别器估计轨迹级密度比,对结果进行截断后作为权重,用于BC目标中优先保留清洁专家行为,同时降低或丢弃受损数据,无需知晓污染机制。理论分析表明,该方法可收敛至干净专家策略,且有限样本误差界与污染率无关。实验构建了涵盖奖励、状态、转移和动作等多种污染方式的评估框架,在连续控制基准上验证,即使在高污染比例下,Weighted BC仍显著优于传统BC、批处理约束Q学习(BCQ)和行为正则化演员-批评家(BRAC)等基线方法。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) enables policy optimization from fixed datasets, making it suitable for safety-critical applications where online exploration is infeasible. However, these datasets are often contaminated by adversarial poisoning, system errors, or low-quality samples, leading to degraded policy performance in standard behavioral cloning (BC) and offline RL methods. This paper introduces Density-Ratio Weighted Behavioral Cloning (Weighted BC), a robust imitation learning approach that uses a small, verified clean reference set to estimate trajectory-level density ratios via a binary discriminator. These ratios are clipped and used as weights in the BC objective to prioritize clean expert behavior while down-weighting or discarding corrupted data, without requiring knowledge of the contamination mechanism. We establish theoretical guarantees showing convergence to the clean expert policy with finite-sample bounds that are independent of the contamination rate. A comprehensive evaluation framework is established, which incorporates various poisoning protocols (reward, state, transition, and action) on continuous control benchmarks. Experiments demonstrate that Weighted BC maintains near-optimal performance even at high contamination ratios outperforming baselines such as traditional BC, batch-constrained Q-learning (BCQ) and behavior regularized actor-critic (BRAC).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。