压缩推理过程可显著提升知识蒸馏效率,同时保持高精度。
Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
- 对长推理链进行后处理压缩,减少蒸馏数据量
- 训练token减少70%-88%,速度提升2-7.6倍,输出缩短3-19倍
- 适合追求高效推理的中小模型部署场景
推理模型生成的思维链过长,导致知识蒸馏成本高且学生模型输出冗余。本文研究在知识蒸馏前对推理链进行后处理压缩。使用Qwen3.5-397B-A17B和gpt-oss-120B两个教师模型各生成约28.3万条正确推理链,再通过两个指令微调模型将其压缩至原字符长度的8.6%-21.0%。在48次主实验及七次针对Qwen教师的截断消融实验中,压缩后的推理链使训练token减少至原始的12%-30%,训练速度提升2.0-7.6倍,推理输出长度缩短3-19倍;但原始推理链在所有规模下仍保持最高下游准确率。长度匹配的原始链截断对比表明,压缩并非仅因预算减小:模型压缩结果通常优于或等于简单截断,尤其在小模型上表现更优,且推理输出更短。总体而言,推理链压缩提供了一个精度-效率权衡:学生模型在保留高达96%原始准确率的同时,每token效率最高提升18倍;在0.8B规模下,经LoRA压缩的链虽缩小了与原始链的差距,但未超越原始链性能。
原文摘要 · Abstract (English)
Reasoning models produce long chain-of-thought traces that are costly to distill and encourage verbose student outputs. We study post-hoc compression of such traces before knowledge distillation. Two teachers, Qwen3.5-397B-A17B and gpt-oss-120B, generate about 283k correct traces each; two instruction-tuned models then compress them to 8.6-21.0% of their original character length. Across a 48-run main grid plus seven Qwen-teacher truncation ablations, compressed traces reduce training tokens to 12-30% of raw, speed up training by 2.0-7.6x, and shorten inference outputs by 3-19x with smaller reductions under the shorter gpt-oss teacher. However, raw traces retain the highest downstream accuracy at every scale and for both teachers. A length-matched raw-trace truncation ablation shows that compression is not merely benefiting from a smaller token budget: model-compressed traces usually beat or match naive truncation, especially for smaller students, while maintaining shorter inference outputs. Overall, reasoning-trace compression offers an accuracy-efficiency trade-off rather than a free improvement: students retain up to 96% of raw-trace accuracy while gaining up to 18x higher per-token efficiency, and at the 0.8B scale under LoRA compressed traces narrow the raw-vs-compressed gap but do not exceed raw.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。