用强化学习优化大模型决策，解决其盲目追求短期收益的问题。
LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
- 通过自生成思维链进行强化学习微调，提升探索能力。
- 在多臂老虎机等任务中，决策准确率显著提高，知行差距缩小。
- 适合研究大模型智能体、强化学习应用的开发者和研究人员。
大语言模型在智能体应用中表现引人关注，但其决策常因贪婪、频率偏差及知行不一而失效。本文系统研究了这些缺陷成因，提出基于自生成思维链的强化学习微调方法，以缓解贪婪性与探索不足问题。在多臂老虎机、上下文老虎机和井字棋任务中，该方法有效提升了探索能力，缩小了模型知识与实际行为间的差距。同时对比了经典探索策略（如ε-贪婪）与模型特有方法（如自修正、自一致性），验证了其对强化学习微调的促进作用。
原文摘要 · Abstract (English)
The success of Large Language Models (LLMs) has sparked interest in various agentic applications. A key hypothesis is that LLMs, leveraging common sense and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently solve complex domains. However, LLM agents have been found to suffer from sub-optimal exploration and the knowing-doing gap, the inability to effectively act on knowledge present in the model. In this work, we systematically study why LLMs perform sub-optimally in decision-making scenarios. In particular, we closely examine three prevalent failure modes: greediness, frequency bias, and the knowing-doing gap. We propose mitigation of these shortcomings by fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales. Our experiments across multi-armed bandits, contextual bandits, and Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. Finally, we study both classic exploration mechanisms, such as $ε$-greedy, and LLM-specific approaches, such as self-correction and self-consistency, to enable more effective fine-tuning of LLMs for decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。