用信念分布增强提示，让Transformer在不更新参数下更好做决策
Heuristic Transformer: Belief Augmented In-Context Reinforcement Learning
- 用变分自编码器学奖励的信念分布，作为提示输入Transformer
- 在Darkroom等3个环境上超越基线，提升决策效果与泛化能力
- 适合想在不训练模型前提下快速适应新任务的研究者
Transformer在上下文学习（ICL）方面表现卓越，已广泛应用于自然语言处理、计算机视觉和序列决策任务。在强化学习中，ICL将学习问题转化为监督任务，实现无需参数更新的任务适应。基于先前将Transformer用于序列决策的工作，我们提出启发式Transformer（HT），一种基于上下文强化学习（ICRL）的方法，通过在上下文数据集中引入对奖励的信念分布，提升决策性能。利用变分自编码器（VAE），学习一个低维随机变量来表示奖励的后验分布，并将其与上下文数据集及查询状态一同作为提示输入Transformer策略。我们在Darkroom、Miniworld和MuJoCo环境中评估了HT的表现，结果表明其在有效性与泛化能力上均持续优于可比基线。该方法为信念增强与Transformer决策之间的桥梁提供了有前景的方向。
原文摘要 · Abstract (English)
Transformers have demonstrated exceptional in-context learning (ICL) capabilities, enabling applications across natural language processing, computer vision, and sequential decision-making. In reinforcement learning, ICL reframes learning as a supervised problem, facilitating task adaptation without parameter updates. Building on prior work leveraging transformers for sequential decision-making, we propose Heuristic Transformer (HT), an in-context reinforcement learning (ICRL) approach that augments the in-context dataset with a belief distribution over rewards to achieve better decision-making. Using a variational auto-encoder (VAE), a low-dimensional stochastic variable is learned to represent the posterior distribution over rewards, which is incorporated alongside an in-context dataset and query states as prompt to the transformer policy. We assess the performance of HT across the Darkroom, Miniworld, and MuJoCo environments, showing that it consistently surpasses comparable baselines in terms of both effectiveness and generalization. Our method presents a promising direction to bridge the gap between belief-based augmentations and transformer-based decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。