不用强化学习，用自蒸馏让大模型更会解长程推理题
Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning
- 同一模型当老师和学生，通过语义纠错自动生成训练数据
- 在GSM8K上微调后，MATH500和AIME2024准确率提升10.6%和10%
- 无需人工标注，适合想提升数学推理能力的模型优化者
大语言模型的长程推理能力可通过思维链（CoT）推理解题得到增强。传统训练方法依赖可验证奖励的强化学习（RLVR），但存在奖励稀疏、样本效率低等问题，需大量计算资源。本文提出语义软自蒸馏（Semantic Soft Bootstrapping, SSB），让同一基础模型同时扮演教师与学生角色，训练时接收关于答案正确性的不同语义上下文。先生成多个解题路径，筛选出正确答案与最常见的错误答案，再将其作为上下文引导模型生成更稳健的逐步解释和验证后的最终答案。该流程自动从原始问题-答案数据中构建师生配对训练集，无需人工干预。生成过程还产出一系列logits，学生模型仅凭原始问题就需匹配这些输出。实验中，Qwen2.5-3B-Instruct在GSM8K上经参数高效微调后，在MATH500和AIME2024基准上分别取得10.6%和10%的准确率提升，优于常用强化学习算法GRPO。代码与模型已开源。
原文摘要 · Abstract (English)
Long context reasoning in large language models (LLMs) has demonstrated enhancement of their cognitive capabilities via chain-of-thought (CoT) inference. Training such models is usually done via reinforcement learning with verifiable rewards (RLVR) in reasoning based problems, like math and programming. However, RLVR is limited by several bottlenecks, such as, lack of dense reward, and inadequate sample efficiency. As a result, it requires significant compute resources in post-training phase. To overcome these limitations, in this work, we propose \textbf{Semantic Soft Bootstrapping (SSB)}, a self-distillation technique, in which the same base language model plays the role of both teacher and student, but receives different semantic contexts about the correctness of its outcome at training time. The model is first prompted with a math problem and several rollouts are generated. From them, the correct and most common incorrect response are filtered, and then provided to the model in context to produce a more robust, step-by-step explanation with a verified final answer. This pipeline automatically curates a paired teacher-student training set from raw problem-answer data, without any human intervention. This generation process also produces a sequence of logits, which is what the student model tries to match in the training phase just from the bare question alone. In our experiment, Qwen2.5-3B-Instruct on GSM8K dataset via parameter-efficient fine-tuning. We then tested its accuracy on MATH500, and AIME2024 benchmarks. Our experiments show a jump of 10.6%, and 10% improvements in accuracy, respectively, over group relative policy optimization (GRPO), which is a commonly used RLVR algorithm. Our code is available at https://github.com/purbeshmitra/semantic-soft-bootstrapping, and the model, curated dataset is available at https://huggingface.co/purbeshmitra/semantic-soft-bootstrapping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。