arXiv:2606.21704cs.CL2026-06

分析推理压缩在不同场景下的效果,给出适配建议。

When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation

论文配图:When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
图 1 · 摘自论文原文
  • 按重要性、重构层级和预算三维度系统评估压缩方法
  • 数学任务需保留结构,通用任务可大幅重写降噪
  • 训练压缩不等于推理节省,长推理模型仍冗余

链式思维(CoT)压缩将大模型的多步推理迁移到小模型,但教师生成的冗长推理路径增加了训练与推理开销。现有压缩方法分为选择性剪枝与生成重写两类,但以往研究未厘清关键因素:剪枝中的粒度与重要性标准混杂,重写中的重构层级常被忽略,压缩预算也缺乏跨领域与场景的系统评估。本文从重要性准则、重构层级、压缩预算三个维度重新审视CoT压缩,覆盖数学与通用两个领域及长/短推理两种范式。结果发现:(i) 重要性准则的有效性由粒度决定——步骤级准则收敛于共享推理主干,而词元级剪枝需依赖符号感知信号以保留逻辑核心;(ii) 重构层级在不同领域呈现反向效应:数学任务随结构破坏单调退化,而通用任务中激进重写可作为去噪手段;(iii) 训练阶段压缩未必带来推理阶段收益:长推理学生即使接受简洁监督,仍保持冗余习惯,因此训练比例仅为部署成本的乐观下界。这些发现为压缩策略与部署场景的匹配提供条件感知指导。

原文摘要 · Abstract (English)

Chain-of-Thought (CoT) distillation transfers multi-step reasoning from large reasoning models to smaller students, but verbose teacher traces inflate both training and inference cost. Existing CoT compression methods fall into two families, selective pruning and generative rewriting, yet prior studies have left key factors entangled: granularity is confounded with importance criteria in pruning, restructuring level is rarely isolated in rewriting, and compression budgets are not systematically evaluated across domains or regimes. We recast CoT compression along three dimensions: importance criterion, restructuring level, and compression budget. Sweeping these across two model families, Math and General domains, and Long-/Short-CoT regimes, we find that (i) importance criterion utility is strictly governed by granularity: step-level criteria converge on a shared reasoning backbone, while token-level pruning requires symbol-aware signals to preserve the logical core; (ii) restructuring level inverts across domains: Math degrades monotonically with structural disruption, while aggressive rewriting acts as a denoiser on General tasks; (iii) training-time compression does not necessarily translate to inference-time savings: Long-CoT students retain verbose habits despite concise supervision, making the training ratio an optimistic lower bound on deployment cost. These findings yield condition-aware guidelines for matching compression to deployment context.

推理压缩链式思维模型蒸馏条件适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。