用小模型指导大模型推理，效率提升90%以上
LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?
- 通过对比大模型与小模型的行为差异，挖掘关键推理时刻
- 在7个数学任务上准确率最高提升28.1%，训练耗时减少90%
- 无需真实标签，适合资源有限但需提升推理能力的研究者
大型语言模型（LLMs）在推理方面取得显著进展，通常依赖监督微调（SFT）。然而SFT成本高昂，需大量精心构建的数据集、拒绝采样示例，并对所有令牌进行统一优化，尽管仅有少量信息具有实际学习价值。本文提出一种反直觉思路：能否让小型语言模型（SLMs）通过揭示反映大模型优势的高价值推理时刻，来指导大模型？我们提出LightReasoner框架，利用强专家模型（LLM）与弱业余模型（SLM）之间的行为差异。该框架分两阶段运行：(1) 采样阶段，识别关键推理时刻并构建体现专家优势的对比性监督样本；(2) 微调阶段，将专家模型对齐这些精炼样本，强化其推理能力。在七个数学基准测试中，LightReasoner准确率最高提升28.1%，时间消耗降低90%，采样问题减少80%，微调令牌使用量下降99%，且不依赖真实标签。通过将弱小模型转化为有效教学信号，LightReasoner提供了一种可扩展、资源高效的增强LLM推理的方法。代码已公开于：https://github.com/HKUDS/LightReasoner
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable progress in reasoning, often through supervised fine-tuning (SFT). However, SFT is resource-intensive, relying on large curated datasets, rejection-sampled demonstrations, and uniform optimization across all tokens, even though only a fraction carry meaningful learning value. In this work, we explore a counterintuitive idea: can smaller language models (SLMs) teach larger language models (LLMs) by revealing high-value reasoning moments that reflect the latter's unique strength? We propose LightReasoner, a novel framework that leverages the behavioral divergence between a stronger expert model (LLM) and a weaker amateur model (SLM). LightReasoner operates in two stages: (1) a sampling stage that pinpoints critical reasoning moments and constructs supervision examples capturing the expert's advantage through expert-amateur contrast, and (2) a fine-tuning stage that aligns the expert model with these distilled examples, amplifying its reasoning strengths. Across seven mathematical benchmarks, LightReasoner improves accuracy by up to 28.1%, while reducing time consumption by 90%, sampled problems by 80%, and tuned token usage by 99%, all without relying on ground-truth labels. By turning weaker SLMs into effective teaching signals, LightReasoner offers a scalable and resource-efficient approach for advancing LLM reasoning. Code is available at: https://github.com/HKUDS/LightReasoner
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。