直接预测信念，减少强化学习延迟中的误差累积。
Directly Forecasting Belief for Reinforcement Learning with Delays
- 跳过逐步推演，直接从观测预测状态
- 实验显示误差显著降低，预测更准确
- 适合处理感知延迟的强化学习场景
带有延迟的强化学习（RL）极具挑战性，因为感官感知滞后于实际事件：RL代理需基于历史观测估算环境真实状态。当前最先进方法通常采用递归式、逐步的状态预测，易导致误差不断累积。本文提出一种新信念估计方法——直接预测信念变压器（DFBT），直接从观测预测状态，不依赖中间步骤的逐步估计。理论证明，与现有递归方法相比，DFBT能大幅减少误差累积，提供更强性能保障。在D4RL离线数据集上的实验表明，DFBT显著降低误差并保持高预测精度。其对状态序列的直接预测能力还支持多步自举，极大提升学习效率。在MuJoCo基准测试中，基于DFBT的方法显著优于现有最先进基线。代码已开源：https://github.com/QingyuanWuNothing/DFBT。
原文摘要 · Abstract (English)
Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based on past observations. State-of-the-art (SOTA) methods typically employ recursive, step-by-step forecasting of states. This can cause the accumulation of compounding errors. To tackle this problem, our novel belief estimation method, named Directly Forecasting Belief Transformer (DFBT), directly forecasts states from observations without incrementally estimating intermediate states step-by-step. We theoretically demonstrate that DFBT greatly reduces compounding errors of existing recursively forecasting methods, yielding stronger performance guarantees. In experiments with D4RL offline datasets, DFBT reduces compounding errors with remarkable prediction accuracy. DFBT's capability to forecast state sequences also facilitates multi-step bootstrapping, thus greatly improving learning efficiency. On the MuJoCo benchmark, our DFBT-based method substantially outperforms SOTA baselines. Code is available at https://github.com/QingyuanWuNothing/DFBT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。