用贝叶斯融合提升上下文强化学习，无需重训练即可快速适应新环境。
In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Prior
- 通过深度集成学习价值先验，测试时用贝叶斯更新融合上下文信息。
- 在仅用次优数据训练下仍能实现近似最优决策，显著降低后悔值。
- 适合需要快速适应新任务且对分布偏移鲁棒的强化学习场景。
上下文强化学习（ICRL）可在不更新参数的情况下快速适应未见环境，但现有方法或无法超越训练分布，或需接近最优数据，限制了实际应用。本文提出SPICE，一种基于贝叶斯融合的ICRL方法：通过深度集成学习Q值先验，并在测试时利用上下文信息进行贝叶斯更新。为应对训练数据不佳导致的劣质先验，其在线推断采用上置信界（UCB）规则，鼓励探索与适应。理论上证明，即使仅在次优轨迹上预训练，SPICE在随机老虎机和有限时域马尔可夫决策过程（MDP）中仍能实现后悔值最优。实验验证了其在老虎机与控制基准上的有效性，显著优于已有ICRL与元强化学习方法，在未见任务上实现近似最优决策，快速适应并保持对分布偏移的鲁棒性。
原文摘要 · Abstract (English)
In-context reinforcement learning (ICRL) promises fast adaptation to unseen environments without parameter updates, but current methods either cannot improve beyond the training distribution or require near-optimal data, limiting practical adoption. We introduce SPICE, a Bayesian ICRL method that learns a prior over Q-values via deep ensemble and updates this prior at test-time using in-context information through Bayesian updates. To recover from poor priors resulting from training on sub-optimal data, our online inference follows an Upper-Confidence Bound rule that favours exploration and adaptation. We prove that SPICE achieves regret-optimal behaviour in both stochastic bandits and finite-horizon MDPs, even when pretrained only on suboptimal trajectories. We validate these findings empirically across bandit and control benchmarks. SPICE achieves near-optimal decisions on unseen tasks, substantially reduces regret compared to prior ICRL and meta-RL approaches while rapidly adapting to unseen tasks and remaining robust under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。