不依赖动作信息，仅用状态和奖励推断任务，提升机器人强化学习泛化能力。
Context Representation via Action-Free Transformer encoder-decoder for Meta Reinforcement Learning
- 用状态与奖励序列直接推断任务表示，无需依赖动作数据。
- 在MetaWorld ML-10上实现更快适应、更好泛化和更高效探索。
- 适合需要模块化训练与可扩展性的机器人控制场景。
强化学习使机器人能在不确定环境中运行，但传统方法常在未见任务上泛化能力差。上下文自适应元强化学习通过任务表示进行条件建模，但大多依赖完整经验中的动作信息，导致任务推断与特定策略紧密耦合。本文提出行动无关的Transformer编码器-解码器模型CRAFT，仅基于状态与奖励序列推断任务表示。通过去除对动作的依赖，CRAFT实现了任务推断与策略优化的解耦，支持模块化训练，并采用近似变分推断实现可扩展的信念更新。模型基于带旋转位置编码的Transformer架构，能捕捉长程时序依赖，稳健编码参数与非参数任务变化。在MetaWorld ML-10机器人操作基准测试中，CRAFT相较上下文自适应元强化学习基线表现出更快适应、更强泛化与更有效探索能力。结果表明，无动作推理为可扩展机器人控制强化学习提供了新路径。
原文摘要 · Abstract (English)
Reinforcement learning (RL) enables robots to operate in uncertain environments, but standard approaches often struggle with poor generalization to unseen tasks. Context-adaptive meta reinforcement learning addresses these limitations by conditioning on the task representation, yet they mostly rely on complete action information in the experience making task inference tightly coupled to a specific policy. This paper introduces Context Representation via Action Free Transformer encoder decoder (CRAFT), a belief model that infers task representations solely from sequences of states and rewards. By removing the dependence on actions, CRAFT decouples task inference from policy optimization, supports modular training, and leverages amortized variational inference for scalable belief updates. Built on a transformer encoder decoder with rotary positional embeddings, the model captures long range temporal dependencies and robustly encodes both parametric and non-parametric task variations. Experiments on the MetaWorld ML-10 robotic manipulation benchmark show that CRAFT achieves faster adaptation, improved generalization, and more effective exploration compared to context adaptive meta--RL baselines. These findings highlight the potential of action-free inference as a foundation for scalable RL in robotic control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。