arXiv:2602.18694cs.LGcs.AI2026-02

用离线数据学习抽象时间动作,实现无需更新的快速规划。

In-Context Planning with Latent Temporal Abstractions

  • 将观测与动作压缩为分层离散符号,构建可自回归预测的隐空间模型。
  • 在多种任务上超越主流基线,尤其在随机与部分可观测场景表现优异。
  • 适合需要快速适应新环境的机器人控制与强化学习应用。

连续控制中的基于规划的强化学习受限于两个实际问题:在原始时间尺度上规划导致分支指数增长和长时程;而真实环境常具有部分可观测性并存在状态切换,破坏了静态完全可观测的假设。我们提出 I-TAP(上下文感知的潜在时间抽象规划器),一个将上下文适应与在线规划统一于学习到的离散时间抽象空间中的离线强化学习框架。从离线轨迹中,I-TAP 学习一个观测条件下的残差量化变分自编码器(VAE),将每个观测-宏观动作片段压缩为粗粒度到细粒度的离散残差标记堆栈,并训练一个时间注意力模型,从近期短历史序列自回归预测这些标记堆栈。所得序列模型同时充当上下文条件的抽象动作先验与潜在动力学模型。测试时,I-TAP 在标记空间中直接执行蒙特卡洛树搜索,利用短期历史实现隐式适应而无需梯度更新,并解码选定的标记堆栈生成可执行动作。在确定性与随机性的 MuJoCo 环境、每轮具有隐含动态状态切换的随机性版本,以及高维的 Adroit 操控任务(包括部分可观测变体)中,I-TAP 均持续匹配或超越强模型无关与模型驱动的离线基线,证明了其在随机动态与部分可观测性下的高效且稳健的上下文规划能力。

原文摘要 · Abstract (English)

Planning-based reinforcement learning for continuous control is bottlenecked by two practical issues: planning at primitive time scales leads to prohibitive branching and long horizons, while real environments are frequently partially observable and exhibit regime shifts that invalidate stationary, fully observed dynamics assumptions. We introduce I-TAP (In-Context Latent Temporal-Abstraction Planner), an offline RL framework that unifies in-context adaptation with online planning in a learned discrete temporal-abstraction space. From offline trajectories, I-TAP learns an observation-conditioned residual-quantization VAE that compresses each observation-macro-action segment into a coarse-to-fine stack of discrete residual tokens, and a temporal Transformer that autoregressively predicts these token stacks from a short recent history. The resulting sequence model acts simultaneously as a context-conditioned prior over abstract actions and a latent dynamics model. At test time, I-TAP performs Monte Carlo Tree Search directly in token space, using short histories for implicit adaptation without gradient update, and decodes selected token stacks into executable actions. Across deterministic MuJoCo, stochastic MuJoCo with per-episode latent dynamics regimes, and high-dimensional Adroit manipulation, including partially observable variants, I-TAP consistently matches or outperforms strong model-free and model-based offline baselines, demonstrating efficient and robust in-context planning under stochastic dynamics and partial observability.

强化学习规划时间抽象离线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。