让大模型学会像人一样思考后再回答，提升各类任务表现
Thinking LLMs: General Instruction Following with Thought Generation
- 通过迭代搜索优化生成思维过程，无需人类标注数据
- 在AlpacaEval和Arena-Hard上性能显著提升
- 不仅推理任务受益，营销、健康等非推理任务也获增益
大模型通常直接回答问题或执行指令，缺乏显式思考过程。对于需要推理与规划的复杂任务，思考至关重要，但也可应用于任何任务。本文提出一种训练方法，使现有大模型在不使用额外人类数据的情况下，具备通用指令遵循中的思考能力。该方法通过迭代搜索与优化，探索可能的思维生成空间，利用判别模型仅基于响应评分并采用偏好优化进行调整。实验表明，该方法在AlpacaEval和Arena-Hard上表现更优，并在非推理类任务如营销、健康和常识知识中也展现出思考带来的收益，覆盖传统推理与问题解决之外的广泛场景。
原文摘要 · Abstract (English)
LLMs are typically trained to answer user questions or follow instructions similarly to how human experts respond. However, in the standard alignment framework they lack the basic ability of explicit thinking before answering. Thinking is important for complex questions that require reasoning and planning -- but can be applied to any task. We propose a training method for equipping existing LLMs with such thinking abilities for general instruction following without use of additional human data. We achieve this by an iterative search and optimization procedure that explores the space of possible thought generations, allowing the model to learn how to think without direct supervision. For each instruction, the thought candidates are scored using a judge model to evaluate their responses only, and then optimized via preference optimization. We show that this procedure leads to superior performance on AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning categories such as marketing, health and general knowledge, in addition to more traditional reasoning & problem-solving tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。