arXiv:2603.19266cs.CLcs.AI2026-03中稿 · ICLR被引 1

让小模型学会深度思考,避免死记硬背。

Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion

  • 用解释性探针迫使学生模型说出推理逻辑,而非记忆答案。
  • 在12个数据集上,性能比零样本提升20.39%,优于现有方法6.02%。
  • 适合追求高效、强泛化能力的小模型部署场景。

将大语言模型的稳健推理能力蒸馏到更小、计算效率更高的学生模型中仍是一个未解决的挑战。尽管近期有进展,但蒸馏后的模型常出现表层模式记忆和泛化能力差的问题。为此,我们提出一种新蒸馏框架,超越简单模仿,旨在注入深层概念理解。该框架包含两项关键创新:首先,解释性反演(EI)生成针对性的“解释性探针”,强制学生模型阐述答案背后的逻辑,而非单纯记忆;其次,解释性GRPO(EXGRPO)采用新型强化学习算法,引入对话结构效用奖励,明确奖励学生在多个探针间保持连贯推理过程。在12个数据集上的广泛评估表明显著提升。以Gemma-7b为学生模型时,本方法平均比零样本性能提高20.39%,比当前最优蒸馏基线提升6.02%。此外,经本方法蒸馏的模型表现出优异训练效率(如仅需10-25%的训练数据即超越原始微调),并在分布外任务上具有强泛化能力。代码已开源:https://github.com/Zhen-Tan-dmml/ExGRPO.git。

原文摘要 · Abstract (English)

Distilling robust reasoning capabilities from large language models (LLMs) into smaller, computationally efficient student models remains an unresolved challenge. Despite recent advances, distilled models frequently suffer from superficial pattern memorization and subpar generalization. To overcome these limitations, we introduce a novel distillation framework that moves beyond simple mimicry to instill a deeper conceptual understanding. Our framework features two key innovations. \underline{\textit{First}}, to address pattern memorization, Explanatory Inversion (EI) generates targeted ``explanatory probes'' that compel the student to articulate the underlying logic behind an answer, rather than just memorizing it. \underline{\textit{Second}}, to improve generalization, Explanatory GRPO (\texttt{EXGRPO}) uses a reinforcement learning algorithm with a novel Dialogue Structure Utility Bonus, which explicitly rewards the student for maintaining a coherent reasoning process across these probes. Extensive evaluations on 12 datasets demonstrate significant improvements. Using Gemma-7b as the student model, our method yields an average \textbf{20.39\%} increase over zero-shot performance and a \textbf{6.02\%} improvement over the state-of-the-art distillation baselines. Moreover, models distilled with our method show remarkable training efficiency (e.g., surpassing vanilla fine-tuning with \textbf{10-25\%} training data) and strong generalization to out-of-distribution tasks. Implementation is released at https://github.com/Zhen-Tan-dmml/ExGRPO.git.

模型蒸馏推理增强强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。