用大模型零样本预测连续环境动态，提升强化学习泛化能力
Zero-shot Model-based Reinforcement Learning using Large Language Models
- 分离状态与控制信号，实现大模型对连续动态的上下文建模
- 在两个场景中验证效果，且不确定性估计更可靠
- 适合研究大模型在强化学习中跨领域应用的学者
大型语言模型（LLMs）的零样本能力正拓展至自然语言处理以外的领域。在强化学习中，尽管LLMs已广泛用于文本环境，但其在连续状态空间中的应用仍不充分。本文探讨如何利用预训练的LLMs，在上下文中预测连续马尔可夫决策过程的动力学。我们识别出多变量数据处理和控制信号融合是限制其部署的关键挑战，并提出解耦上下文学习（DICL）方法加以解决。通过理论分析，我们在两种强化学习设置中进行了概念验证：基于模型的策略评估与数据增强的离线策略强化学习。实验表明，该方法能生成校准良好的不确定性估计。代码已开源于 https://github.com/abenechehab/dicl。
原文摘要 · Abstract (English)
The emerging zero-shot capabilities of Large Language Models (LLMs) have led to their applications in areas extending well beyond natural language processing tasks. In reinforcement learning, while LLMs have been extensively used in text-based environments, their integration with continuous state spaces remains understudied. In this paper, we investigate how pre-trained LLMs can be leveraged to predict in context the dynamics of continuous Markov decision processes. We identify handling multivariate data and incorporating the control signal as key challenges that limit the potential of LLMs' deployment in this setup and propose Disentangled In-Context Learning (DICL) to address them. We present proof-of-concept applications in two reinforcement learning settings: model-based policy evaluation and data-augmented off-policy reinforcement learning, supported by theoretical analysis of the proposed methods. Our experiments further demonstrate that our approach produces well-calibrated uncertainty estimates. We release the code at https://github.com/abenechehab/dicl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。