提出新指标量化强化学习中策略对历史信息的依赖程度。
Quantifying Memory Use in Reinforcement Learning with Temporal Range
- 用反向自动微分计算输出对输入的时间影响分布，求加权平均延迟。
- 在多种任务和模型上验证：未观测任务依赖更长历史，任务真实延迟决定记忆需求。
- 可帮助选择最短有效上下文，适合研究记忆机制与模型设计者使用。
训练好的强化学习策略究竟多大程度依赖过去观测？我们提出一种模型无关的指标——时序范围（Temporal Range），将多个输出对时间窗口内输入序列的一阶敏感性视为时序影响分布，并以加权平均滞后来总结。该指标通过反向自动微分计算雅可比块∂yₛ/∂xₜ∈ℝᶜˣᵈ，对最终时间步s∈{t+1,…,T}取平均，在线性情形下由一组自然公理良好刻画。在诊断与控制任务（POPGym；闪烁/遮挡；Copy-k）及架构（MLPs、RNNs、SSMs）中，时序范围（i）在完全可观测控制中保持较小，（ii）随Copy-k任务的真实延迟而增长，（iii）与近优回报所需的最小历史窗口一致，经窗口消融验证。还报告了在任务上训练的紧凑长表达记忆（LEM）策略的时序范围，作为任务级记忆的代理读出。其公理化处理借鉴近期范围度量研究，专用于时序滞后并扩展至强化学习中的向量输出。时序范围为比较智能体与环境、选择最短必要上下文提供了实用的逐序列记忆依赖读出。
原文摘要 · Abstract (English)
How much does a trained RL policy actually use its past observations? We propose \emph{Temporal Range}, a model-agnostic metric that treats first-order sensitivities of multiple vector outputs across a temporal window to the input sequence as a temporal influence profile and summarizes it by the magnitude-weighted average lag. Temporal Range is computed via reverse-mode automatic differentiation from the Jacobian blocks $\partial y_s/\partial x_t\in\mathbb{R}^{c\times d}$ averaged over final timesteps $s\in\{t+1,\dots,T\}$ and is well-characterized in the linear setting by a small set of natural axioms. Across diagnostic and control tasks (POPGym; flicker/occlusion; Copy-$k$) and architectures (MLPs, RNNs, SSMs), Temporal Range (i) remains small in fully observed control, (ii) scales with the task's ground-truth lag in Copy-$k$, and (iii) aligns with the minimum history window required for near-optimal return as confirmed by window ablations. We also report Temporal Range for a compact Long Expressive Memory (LEM) policy trained on the task, using it as a proxy readout of task-level memory. Our axiomatic treatment draws on recent work on range measures, specialized here to temporal lag and extended to vector-valued outputs in the RL setting. Temporal Range thus offers a practical per-sequence readout of memory dependence for comparing agents and environments and for selecting the shortest sufficient context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。