让大模型更耐心推理，用少量数据提升解题准确率
Patience Is The Key to Large Language Model Reasoning
- 用正负样本对比训练模型偏好详细推理过程
- 在GSM8k上仅用轻量数据就提升2.1%准确率
- 无需新知识或复杂训练，适合资源有限场景
近期大型语言模型通过思维链（Chain of Thought, CoT）方法在解决复杂问题上取得显著进展。然而，现有模型或因用户偏好而牺牲推理深度，或需大量昂贵训练数据才能掌握复杂推理能力，限制了其在复杂任务中的应用。为此，本文受测试时扩展（scaling test-time）思想启发，提出一种简单方法：通过鼓励模型采用更耐心的推理风格，无需引入新知识或技能即可提升表现。具体地，利用偏好优化策略，生成详尽推理过程作为正例，简单答案作为负例，训练模型更倾向于完整推理。实验结果表明，仅在轻量数据集上训练，模型在GSM8k上的性能最高提升2.1%。
原文摘要 · Abstract (English)
Recent advancements in the field of large language models, particularly through the Chain of Thought (CoT) approach, have demonstrated significant improvements in solving complex problems. However, existing models either tend to sacrifice detailed reasoning for brevity due to user preferences, or require extensive and expensive training data to learn complicated reasoning ability, limiting their potential in solving complex tasks. To bridge this gap, following the concept of scaling test-time, we propose a simple method by encouraging models to adopt a more patient reasoning style without the need of introducing new knowledge or skills. To employ a preference optimization approach, we generate detailed reasoning processes as positive examples and simple answers as negative examples, thereby training the model to favor thoroughness in its responses. Our results demonstrate a performance increase of up to 2.1% on GSM8k with training just on a lightweight dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。