非线性Transformer通过共享权重实现跨领域强化学习泛化。
One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning
- 将Transformer视为核空间中的回归器，实现跨域值函数共享。
- 在多领域MetaWorld上验证了时序差分目标的收敛性。
- 适合关注跨领域自适应与模型泛化的研究者。
强化学习的核心挑战是学习能在训练任务之外泛化的模型，传统方法依赖多任务和元强化学习。近期，Transformer架构通过上下文学习实现新任务适应，无需显式参数更新。从功能上看，Transformer可被视为将上下文映射到特定任务函数的算子，因此设计该算子对提升强化学习泛化能力至关重要。本文从核方法视角出发，建立非线性Transformer与基于核的时序差分学习之间的联系。通过将Transformer解释为在再生核希尔伯特空间（RKHS）中的回归操作，我们证明：只要不同领域的值函数位于同一RKHS内，即可使用共享权重进行表示。多个MetaWorld领域的实验支持这一观点，展示了时序差分目标的收敛性。
原文摘要 · Abstract (English)
A central challenge in reinforcement learning (RL) is to learn models that generalize beyond the tasks on which they are trained, a goal traditionally pursued through multi-task and meta RL. Recently, transformer architectures have emerged as a promising approach, enabling adaptation to new tasks via in-context learning without explicit parameter updates. From a functional perspective, a transformer can be viewed as a functional operator that maps a context to a task-specific function. It is thus fundamental to understand and design this operator to support stronger generalization in RL. In this work, we address this resulting question of generalization from a kernel-based perspective by establishing a connection between non-linear transformers and kernel-based temporal difference learning. By interpreting the transformer as performing regression in a Reproducing Kernel Hilbert Space (RKHS), we show that value functions from different domains can be represented using a shared set of weights, provided they lie within the same RKHS. Experiments on multiple MetaWorld domains support this interpretation, demonstrating convergence of the temporal-difference objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。