用强化学习训练大模型下棋，发现其战略思维仍远不如专家。
Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess
- 用棋局预训练的估值网络为模型输出提供密集奖励，实现知识蒸馏。
- 密集奖励比稀疏奖励效果更好，但模型仍无法达到专家水平。
- 问题根源在于模型对棋局理解不足，仅靠强化学习难弥补缺陷。
尽管强化学习（RL）在提升大语言模型（LLM）数学推理能力方面展现潜力，但利用RL发展LLM的战略推理能力仍基本未被探索。本文研究了通过强化学习在国际象棋中能否使LLM具备战略推理能力。为此，我们采用预训练的棋局估值网络，为模型输出走法的质量提供密集奖励，可视为一种知识蒸馏。实验表明，基于蒸馏的密集奖励通常优于稀疏的二值奖励。然而令人意外的是，所有模型的表现均显著低于专家水平。通过对监督微调（SFT）和强化学习的消融分析，我们发现这一局限源于预训练模型对棋局内在理解的不足，而仅靠强化学习难以完全克服。代码已开源：https://github.com/krafton-ai/Chess-R1。
原文摘要 · Abstract (English)
While reinforcement learning (RL) for large language models (LLMs) has shown promise in mathematical reasoning, strategic reasoning for LLMs using RL remains largely unexplored. We investigate whether LLMs can develop strategic reasoning capabilities through RL in chess. To this end, we leverage a chess-pretrained action-value network to provide dense reward on the LLM's output move quality, which can be seen as a form of knowledge distillation. Our experiments show that our distillation-based dense rewards often outperform sparse binary rewards. However, surprisingly, all models plateau far below expert levels. We provide SFT and RL ablations on chess reasoning training and find evidence that this limitation stems from a deficit in the pretrained models' internal understanding of chess-a deficit which RL alone may not be able to fully overcome. The code is available at https://github.com/krafton-ai/Chess-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。