让小模型逐步学会内化符号知识，提升推理能力并降低计算开销。
$\textit{SKIntern}$: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models
- 通过渐进式微调和课程学习，让小模型逐步内化符号知识。
- 在跨领域任务中性能超主流方法5%以上，推理耗时减少4倍。
- 适合追求高效推理的小模型部署场景，如移动端或隐私敏感应用。
小型语言模型（SLMs）因大模型的高算力需求和隐私问题而受到关注。现有研究通过从大模型蒸馏思维链（CoT）数据来增强小模型的推理能力。部分方法引入外部符号知识以弥补小模型知识记忆不足、推理能力弱及跨领域泛化差的问题，但会增加计算开销并引入噪声。本文提出SKIntern，一种创新方法，使小模型通过渐进式微调与预设线性衰减调度的课程学习，逐步内化符号知识与少量示例。该方法有效减少计算开销，推理时仅需关注问题，显著提升效率。在多种小模型上，其性能超过当前最优基线超5%，推理成本（以FLOPs计）最高降低4倍，适用于域内与跨域任务。
原文摘要 · Abstract (English)
Small Language Models (SLMs) are attracting attention due to the high computational demands and privacy concerns of Large Language Models (LLMs). Some studies fine-tune SLMs using Chains of Thought (CoT) data distilled from LLMs, aiming to enhance their reasoning ability. Furthermore, Some CoT distillation methods introduce external symbolic knowledge into the generation process to improve the limited knowledge memory, reasoning ability and out-of-domain (OOD) generalization of SLMs. However, the introduction of symbolic knowledge increases computational overhead and introduces potential noise. In this paper, we introduce $\textit{SKIntern}$, an innovative approach that empowers SLMs to internalize symbolic knowledge and few-shot examples gradually through a progressive fine-tuning process, guided by a predefined linear decay schedule under curriculum learning. By efficiently internalizing knowledge, $\textit{SKIntern}$ reduces computational overhead and speeds up the reasoning process by focusing solely on the question during inference. It outperforms state-of-the-art baselines by over 5\%, while reducing inference costs (measured in FLOPs) by up to $4\times$ across a wide range of SLMs in both in-domain (ID) and out-of-domain (OOD) tasks. Our code will be available at \url{https://github.com/Xnhyacinth/SKIntern}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。