arXiv:2603.05433cs.LG2026-03被引 41

让大模型自己教自己更简洁地推理,减少冗余输出。

CRISP: Compressed Reasoning via Iterative Self-Policy Distillation

  • 用模型自身生成的简洁回答作为教师信号,反向优化学生模型。
  • 在MATH-500上推理长度减少56%,准确率提升3.3点;AIME 2024上减少38%长度,准确率几乎不变。
  • 无需标注答案或预算,适配不同模型规模和架构,通用性强。

推理模型常生成远超任务需求的词元,导致推理成本高且错误累积。本文提出CRISP(基于迭代自策略蒸馏的压缩推理),一种在线自蒸馏方法,通过将模型自身的简洁行为回授自身,实现更紧凑的推理。核心思想是:对同一模型施加“请简洁”指令获取教师词元分布,再在学生模型自身的生成轨迹上最小化逐词元反向KL散度。该方法无需真实答案、词元预算或难度估计器。反向KL目标天然具备难度自适应性:对简单问题激进压缩,对复杂问题保留必要推理步骤。在Qwen3-14B上,CRISP使MATH-500推理长度缩短56%,准确率提升最多3.3点;AIME 2024上长度减少38%,准确率下降不足1点。该效果在不同模型规模与类型间具有泛化能力:Qwen3-8B实现相同压缩且准确率无损,DeepSeek-R1-Distill-Llama-8B在五项基准上同时提升准确率并缩短输出。所有模型的通用能力均保持不变。代码已开源。

原文摘要 · Abstract (English)

Reasoning models often generate far more tokens than a task requires, which raises inference cost and can compound errors. We introduce CRISP (Compressed Reasoning via Iterative Self-Policy Distillation), an on-policy self-distillation method that teaches a model to reason more concisely by distilling its own concise behavior back into itself. The method uses a single idea: condition the same model on a "be concise" instruction to obtain teacher logits, then minimize the per-token reverse KL divergence between the student and this teacher on the student's own rollouts. It requires no ground-truth answers, no token budgets, and no difficulty estimators. The reverse-KL objective is naturally difficulty-adaptive: it compresses easy problems aggressively while preserving the reasoning steps that hard problems require. On Qwen3-14B, CRISP cuts reasoning length by up to 56% on MATH-500 and 38% on the harder AIME 2024, while improving MATH-500 accuracy by up to 3.3 points over the base model and holding AIME 2024 accuracy within about one point. This behavior generalizes across model sizes and families: Qwen3-8B shows the same compression with accuracy preserved, and DeepSeek-R1-Distill-Llama-8B improves accuracy on all five benchmarks while shortening its responses. General capabilities are preserved across all three models. Code is available at https://github.com/HJSang/OPSD_Reasoning_Compression.

推理压缩自蒸馏大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。