用紧凑不确定集替代大模型集成，提升离线强化学习鲁棒性
Epistemic Robust Offline Reinforcement Learning
- 用紧凑不确定性集合替代传统集成，更高效建模认知不确定性
- 在表格与连续状态空间中均优于基于集成的基线方法
- 适合关注算法鲁棒性与数据偏差问题的研究者
离线强化学习从固定数据集学习策略，无需与环境进一步交互。其关键挑战是认知不确定性，源于数据覆盖有限或有偏，尤其当行为策略系统性避开某些动作时，会导致价值估计不准确和泛化不可靠。现有基于集成的方法如SAC-N通过取集成最小值保守估计Q值，但需大规模集成，且常混淆认知与随机不确定性。为此，本文提出统一可推广的框架，将离散集成替换为对Q值的紧凑不确定性集合。我们进一步引入Epinet模型，直接优化不确定性集合以实现累积奖励最大化，无需依赖集成。还构建了一个评估风险敏感行为策略下离线RL算法的基准，并证明该方法在表格与连续状态域中均显著提升鲁棒性与泛化能力。
原文摘要 · Abstract (English)
Offline reinforcement learning learns policies from fixed datasets without further environment interaction. A key challenge in this setting is epistemic uncertainty, arising from limited or biased data coverage, particularly when the behavior policy systematically avoids certain actions. This can lead to inaccurate value estimates and unreliable generalization. Ensemble-based methods like SAC-N mitigate this by conservatively estimating Q-values using the ensemble minimum, but they require large ensembles and often conflate epistemic with aleatoric uncertainty. To address these limitations, we propose a unified and generalizable framework that replaces discrete ensembles with compact uncertainty sets over Q-values. %We further introduce an Epinet based model that directly shapes the uncertainty sets to optimize the cumulative reward under the robust Bellman objective without relying on ensembles. We also introduce a benchmark for evaluating offline RL algorithms under risk-sensitive behavior policies, and demonstrate that our method achieves improved robustness and generalization over ensemble-based baselines across both tabular and continuous state domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。