用大模型先验提升强化学习效率，探索次数减少90%以上
Efficient Reinforcement Learning with Large Language Model Priors
- 将大模型作为动作先验，通过贝叶斯推断融入强化学习框架
- 在离线学习中样本效率提升超90%，显著降低探索成本
- 适用于需要快速决策的复杂任务，尤其适合资源受限场景
在序列决策任务中，强化学习和启发式搜索虽取得进展，但常需大量探索，且泛化能力受限。大语言模型（LLM）具备丰富的领域知识，可作为先验。本文提出将LLM视为动作先验，通过变分推断与直接后验采样融入基于策略和基于价值的强化学习框架。实验表明，使用LLM先验可大幅减少探索与优化复杂度，在离线学习中样本需求降低超过90%。
原文摘要 · Abstract (English)
In sequential decision-making (SDM) tasks, methods like reinforcement learning (RL) and heuristic search have made notable advances in specific cases. However, they often require extensive exploration and face challenges in generalizing across diverse environments due to their limited grasp of the underlying decision dynamics. In contrast, large language models (LLMs) have recently emerged as powerful general-purpose tools, due to their capacity to maintain vast amounts of domain-specific knowledge. To harness this rich prior knowledge for efficiently solving complex SDM tasks, we propose treating LLMs as prior action distributions and integrating them into RL frameworks through Bayesian inference methods, making use of variational inference and direct posterior sampling. The proposed approaches facilitate the seamless incorporation of fixed LLM priors into both policy-based and value-based RL frameworks. Our experiments show that incorporating LLM-based action priors significantly reduces exploration and optimization complexity, substantially improving sample efficiency compared to traditional RL techniques, e.g., using LLM priors decreases the number of required samples by over 90% in offline learning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。