arXiv:2509.01793cs.LGcs.AI2025-09被引 1

构建首个系统性评估随机环境的基准,揭示当前强化学习算法在真实场景中的脆弱性。

STORI: A Benchmark and Taxonomy for Stochastic Environments

  • 提出五类环境随机性分类,涵盖动作干扰、观测噪声等复杂不确定性。
  • 实验显示顶尖模型如DreamerV3在部分可观测下动态预测不可靠,方差估计严重不足。
  • 适合研究鲁棒强化学习、真实世界部署的算法开发者使用。

强化学习(RL)在Atari100k等仿真基准上表现优异,但近期进展多局限于模拟环境,难以迁移到真实世界。核心障碍是环境随机性:真实系统存在观测噪声、动态不可预测和非平稳条件,破坏现有方法的稳定性。现有基准通常忽略这些不确定性,偏好可调优的简化设置,且缺乏对随机性的系统分类,导致对某一类扰动(如粘滞动作)鲁棒的算法未必能应对其他类型。为此,我们提出STORI(STOchastic-ataRI),一个系统整合多种随机效应的基准,支持在不同不确定性形式下严谨评估RL技术。我们建立五类环境随机性分类体系,并通过针对DreamerV3和STORM的定向评估,揭示了当前主流模型基础型方法的系统性弱点:世界模型显著低估环境方差,难以应对动作污染,在部分可观测下表现出不可靠的动力学建模能力。代码与基准已公开于https://github.com/ARY2260/stori,为开发更鲁棒的强化学习系统提供统一框架。

原文摘要 · Abstract (English)

Reinforcement learning (RL) techniques have achieved impressive performance on simulated benchmarks such as Atari100k, yet recent advances remain largely confined to simulation and show limited transfer to real-world domains. A central obstacle is environmental stochasticity, as real systems involve noisy observations, unpredictable dynamics, and non-stationary conditions that undermine the stability of current methods. Existing benchmarks rarely capture these uncertainties and favor simplified settings where algorithms can be tuned to succeed. The absence of a well-defined taxonomy of stochasticity further complicates evaluation, as robustness to one type of stochastic perturbation, such as sticky actions, does not guarantee robustness to other forms of uncertainty. To address this critical gap, we introduce STORI (STOchastic-ataRI), a benchmark that systematically incorporates diverse stochastic effects and enables rigorous evaluation of RL techniques under different forms of uncertainty. We propose a comprehensive five-type taxonomy of environmental stochasticity and demonstrate systematic vulnerabilities in state-of-the-art model-based RL algorithms through targeted evaluation of DreamerV3 and STORM. Our findings reveal that world models dramatically underestimate environmental variance, struggle with action corruption, and exhibit unreliable dynamics under partial observability. We release the code and benchmark publicly at https://github.com/ARY2260/stori, providing a unified framework for developing more robust RL systems.

强化学习随机环境基准测试鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。