通过分解问题，让小模型学会大模型的推理能力。
Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition
- 用问题分解法构建提示，引导小模型模仿大模型思考过程。
- 在GSM8K上小模型准确率提升113%，MATH上提升21%。
- 适合想低成本训练高效推理模型的研究者和开发者。
知识蒸馏使小型神经网络能在计算成本更低的情况下模仿大型教师模型的表现。传统大语言模型的知识蒸馏方法通常需要大量微调，限制了其可及性。为此，我们提出一种新框架——思维链提示（Trace-of-Thought Prompting），旨在将高资源教师模型（超过80亿参数）的关键推理能力蒸馏到低资源学生模型（最高80亿参数）中。该方法利用问题分解提升可解释性，并支持人工干预。在GSM8K和MATH数据集上的实证评估表明，学生模型在GSM8K上准确率最高提升113%，在MATH上提升21%，尤其在Llama 2和Zephyr等较小模型上表现显著。结果表明，开源、低资源模型未来有望兼具学生与教师角色，减少对高资源专有模型的依赖。
原文摘要 · Abstract (English)
Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。