在线强化学习中应对丢帧问题，提升模型鲁棒性。
OnDeFog: Online Decision Transformer under Frame Dropping
- 将离线抗丢帧机制融入在线决策变压器，实现动态适应。
- 在高丢帧率环境下性能优于传统在线方法，且在低奖励数据上表现更优。
- 适合真实场景中通信不稳定、传感器易故障的强化学习任务。
在现实世界强化学习应用中，通信延迟或传感器故障常导致帧丢失，使智能体无法获取丢失的状态及对应奖励。为缓解帧丢失带来的性能下降，已有方法DeFog通过向决策变压器引入额外机制来应对随机帧丢失。然而，由于DeFog是离线学习方法，在训练数据未充分覆盖的新状态上泛化能力有限。本文提出OnDeFog，将DeFog的机制与在线决策变压器（ODT）结合，采用直接环境交互的方式在线学习策略。全面实验表明，相较于ODT，OnDeFog在高丢帧率环境中表现更优；同时在包含大量低奖励数据的数据集上，其性能也超越DeFog。
原文摘要 · Abstract (English)
In challenging real-world reinforcement learning applications, communication delays or sensor failures often cause frame dropping, in which the agent cannot receive the dropped states and associated rewards. To address the performance degradation caused by frame dropping, the Decision Transformer under Random Frame Dropping (DeFog) was developed by incorporating additional mechanisms into the decision transformer to tackle frame dropping. Although DeFog can mitigate performance degradation in frame-dropping environments, since DeFog is an offline learning method, it struggles to effectively generalize to novel states not adequately represented in the training dataset. In this study, we propose OnDeFog, which integrates the mechanisms in DeFog with the online decision transformer (ODT), an online reinforcement learning method that learns policies through direct environmental interaction. Comprehensive experimental evaluation demonstrates that our proposed OnDeFog achieves superior performance compared to ODT in environments characterized by high dropping frame rate and outperforms DeFog on datasets containing a large amount of low-reward data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。