在线强化学习中实现鲁棒性,无需预存数据或生成模型。
Online Robust Reinforcement Learning with General Function Approximation
- 通过双重驱动的拟合贝尔曼算法,边交互边学鲁棒策略。
- 理论证明后悔界为次线性,不随状态动作空间扩大而增长。
- 适合真实场景中环境变化大、无法提前收集数据的强化学习任务。
在许多现实场景中,强化学习系统在部署时若遇到与训练阶段不同的环境,性能会下降。分布鲁棒强化学习(DR-RL)通过寻找在给定不确定性集内最恶劣转移动态下的最优策略来缓解此问题。然而,现有大多数DR-RL方法依赖强数据假设,如需生成模型或大量离线数据,且主要局限于表格型设置。本文提出一种完全在线的DR-RL算法,支持通用函数逼近,仅通过与环境交互即可学习鲁棒策略,无需先验知识或预收集数据。该方法基于双驱动拟合鲁棒贝尔曼过程,同时估计价值函数和对应的最坏情况备份算子。我们建立了在线DR-RL的后悔率保证,其复杂度由鲁棒贝尔曼-艾卢德维维度定义,覆盖广泛的φ-散度不确定性集。所得后悔界为次线性,不随状态或动作空间大小增长,并在结构化问题类中达到紧致速率,表明该框架具有实用性与可扩展性。
原文摘要 · Abstract (English)
In many real-world settings, reinforcement learning systems suffer performance degradation when the environment encountered at deployment differs from that observed during training. Distributionally robust reinforcement learning (DR-RL) mitigates this issue by seeking policies that maximize performance under the most adverse transition dynamics within a prescribed uncertainty set. Most existing DR-RL approaches, however, rely on strong data availability assumptions, such as access to a generative model or large offline datasets, and are largely restricted to tabular settings. In this work, we propose a fully online DR-RL algorithm with general function approximation that learns robust policies solely through interaction, without requiring prior knowledge or pre-collected data. Our approach is based on a dual-driven fitted robust Bellman procedure that simultaneously estimates the value function and the corresponding worst-case backup operator. We establish regret guarantees for online DR-RL characterized by an intrinsic complexity notion, the robust Bellman-Eluder dimension, covering a broad class of phi-divergence uncertainty sets. The resulting regret bounds are sublinear, do not scale with the size of the state or action spaces, and specialize to tight rates in structured problem classes, demonstrating the practicality and scalability of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。