让离线训练的智能体在延迟环境中仍能稳定执行。
Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
- 用Transformer构建信念预测器，提前推断延迟下的真实状态
- 在多种延迟设置下优于基线方法，且更节省数据样本
- 适合需高可靠性、难采集新数据的工业级控制场景
离线强化学习智能体从仿真部署到现实时面临两大挑战：一是模拟与现实间的差距，现实系统存在延迟等仿真中未建模的缺陷；二是交互差距，纯离线训练的策略在在线执行时会遭遇分布外状态，而新数据采集成本高或有风险。因此，智能体必须从静态、无延迟的数据集中泛化到动态、有延迟的环境。标准离线强化学习在无延迟日志上训练，却要在破坏马尔可夫性的延迟环境中行动，性能下降。本文提出DT-CORL（Delay-Transformer belief policy Constrained Offline RL），一种专为应对部署延迟设计的离线强化学习框架。该方法（i）利用基于Transformer的信念预测器，在训练中从未见过延迟观测的情况下生成抗延迟的动作；（ii）相比简单的历史增强基线，显著提升样本效率。在D4RL基准上，多个延迟设置下的实验表明，DT-CORL持续优于历史增强和传统信念方法，在缩小模拟到现实的延迟差距的同时保持数据效率。
原文摘要 · Abstract (English)
Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than naïve history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。