构建可调控的合成环境，精准测试强化学习中记忆模型的能力
Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling
- 基于记忆需求结构理论，设计可定制的POMDP环境
- 通过线性动态与状态聚合生成具有指定记忆挑战的场景
- 提供轻量级可扩展环境，适合评估不同记忆架构
近期面向记忆增强型强化学习（RL）的基准测试引入了部分可观测马尔可夫决策过程（POMDP）环境，要求智能体利用历史观测做出决策。然而，这些基准往往缺乏对记忆模型挑战的精细控制。合成环境提供了解决方案，使研究者能够精确操控环境动态，实现严谨且可解释的评估。本文在该方向上提出三项关键贡献：(1) 构建基于记忆需求结构（MDS）及相关概念的理论分析框架；(2) 提出一种方法，通过线性动力学、状态聚合和奖励重分配来构造具备预设MDS的POMDP；(3) 开发一套轻量、可扩展的POMDP环境，其难度可调，建立在上述理论基础上。总体而言，本工作厘清了部分可观测强化学习中的核心挑战，为POMDP设计提供了原则性指导，并有助于选择和开发适用于特定任务的记忆架构。
原文摘要 · Abstract (English)
Recent benchmarks for memory-augmented reinforcement learning (RL) have introduced partially observable Markov decision process (POMDP) environments in which agents must use historical observations to make decisions. However, these benchmarks often lack fine-grained control over the challenges posed to memory models. Synthetic environments offer a solution, enabling precise manipulation of environment dynamics for rigorous and interpretable evaluation of memory-augmented RL. This paper advances the design of such customizable POMDPs with three key contributions: (1) a theoretical framework for analyzing POMDPs based on Memory Demand Structure (MDS) and related concepts; (2) a methodology using linear dynamics, state aggregation, and reward redistribution to construct POMDPs with predefined MDS; and (3) a suite of lightweight, scalable POMDP environments with tunable difficulty, grounded in our theoretical insights. Overall, our work clarifies core challenges in partially observable RL, offers principled guidelines for POMDP design, and aids in selecting and developing suitable memory architectures for RL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。