让大模型学会下围棋,达到职业水平。
Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Go
- 用围棋专家知识和思维链数据混合微调,再通过强化学习融合。
- 模型能自然语言下棋,预测步法准确,表现媲美职业选手。
- 适合研究通用大模型在专业领域应用的学者与开发者。
大型语言模型(LLMs)在数学、编程等推理任务中表现出色,甚至超越人类。但在专业领域如围棋中,主流LLMs仍无法达到初学者水平,更难以进行自然语言推理。为弥合通用推理能力与领域专长之间的差距,本文提出一种混合微调与强化学习相结合的方法:先以结构化围棋知识和通用思维链数据进行冷启动微调,再引入专家知识增强推理能力。由此构建的LoGos模型不仅保持强大的通用推理能力,还能以自然语言进行围棋对弈,展现出有效的战略思考和精准的下一步预测。其表现接近人类职业选手,显著优于现有所有LLMs。我们还将发布首个大规模围棋用于训练的语料库、首个针对大模型的围棋评估基准,以及首个达到人类职业水平的通用大模型,相关资源见:https://github.com/Entarochuan/LoGos。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated exceptional performance in reasoning tasks such as mathematics and coding, matching or surpassing human capabilities. However, these impressive reasoning abilities face significant challenges in specialized domains. Taking Go as an example, although AlphaGo has established the high performance ceiling of AI systems in Go, mainstream LLMs still struggle to reach even beginner-level proficiency, let alone perform natural language reasoning. This performance gap between general-purpose LLMs and domain experts is significantly limiting the application of LLMs on a wider range of domain-specific tasks. In this work, we aim to bridge the divide between LLMs' general reasoning capabilities and expert knowledge in domain-specific tasks. We perform mixed fine-tuning with structured Go expertise and general long Chain-of-Thought (CoT) reasoning data as a cold start, followed by reinforcement learning to integrate expert knowledge in Go with general reasoning capabilities. Through this methodology, we present \textbf{LoGos}, a powerful LLM that not only maintains outstanding general reasoning abilities, but also conducts Go gameplay in natural language, demonstrating effective strategic reasoning and accurate next-move prediction. LoGos achieves performance comparable to human professional players, substantially surpassing all existing LLMs. Through this work, we aim to contribute insights on applying general LLM reasoning capabilities to specialized domains. We will release the first large-scale Go dataset for LLM training, the first LLM Go evaluation benchmark, and the first general LLM that reaches human professional-level performance in Go at: https://github.com/Entarochuan/LoGos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。