用强化学习微调的Transformer能自学会解新问题，成为通用求解器。
RL + Transformer = A General-Purpose Problem Solver
- 用强化学习微调预训练Transformer，实现上下文内自学习。
- 在未见过的环境中样本效率高，且对分布外数据表现稳健。
- 适合需要自适应、泛化强的复杂任务求解场景。
如果人工智能不仅能解决训练过的任务，还能自我学习解决新问题（即元学习）会怎样？本研究证明，经过强化学习在多轮次中微调的预训练Transformer，展现出一种称为上下文内强化学习（ICRL）的涌现能力：可解决从未遇见过的新问题。该元学习模型不仅在未见的同分布环境中表现出极高的样本效率，还在分布外环境中表现优异。此外，它对训练数据质量具有鲁棒性，能无缝整合上下文中的行为模式，并适应非平稳环境。这些特性表明，经强化学习训练的Transformer能迭代优化自身解法，具备成为通用问题求解器的强大潜力。
原文摘要 · Abstract (English)
What if artificial intelligence could not only solve problems for which it was trained but also learn to teach itself to solve new problems (i.e., meta-learn)? In this study, we demonstrate that a pre-trained transformer fine-tuned with reinforcement learning over multiple episodes develops the ability to solve problems that it has never encountered before - an emergent ability called In-Context Reinforcement Learning (ICRL). This powerful meta-learner not only excels in solving unseen in-distribution environments with remarkable sample efficiency, but also shows strong performance in out-of-distribution environments. In addition, we show that it exhibits robustness to the quality of its training data, seamlessly stitches together behaviors from its context, and adapts to non-stationary environments. These behaviors demonstrate that an RL-trained transformer can iteratively improve upon its own solutions, making it an excellent general-purpose problem solver.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。