通过自适应调度提升小模型推理能力，解决知识蒸馏三大痛点。
CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation
- 按令牌动态调整正反向KL损失，避免初始崩溃。
- 在GSM8K上达69.8%准确率，3B教师下闭合76.2%性能差距。
- 适合追求高效推理的模型压缩研究者。
在线策略知识蒸馏将大模型推理能力迁移到小模型，但现有方法存在三种叠加失败：（i）冷启动坍塌，即新学生对教师偏好标记分配近零概率；（ii）状态无关的发散调度，仅依赖时间的前/后向KL插值忽略学生覆盖状态；（iii）二元奖励稀疏，通过/失败信号丢弃部分正确轨迹信息。本文提出CADENCE框架，针对每种问题提供统一解决方案。其核心为DRIFT机制，在学生采样轨迹上按令牌构建前向与反向KL的凸混合目标（非序列级梯度估计）。六项扩展包括：(A) COVA，基于覆盖率自适应β调度，加速前向到反向过渡；(B) FTB，分叉标记增强，通过全局归一化熵参考聚焦高熵位置梯度；(C) CCD，密集奖励，对错误但接近的轨迹给予数值接近性部分得分；(D) LAP，优先简短正确轨迹的强化；(E) EMR，熵匹配校准正则项；(F) BSD，自举自蒸馏阶段。在GSM8K和MATH-500（修正512令牌协议，5次种子，报告标准差）上，CADENCE将0.5B学生从1.5B教师蒸馏至69.8±0.5% GSM8K pass@1（预训练48.7%；闭合教师差距63.2%），使用3B教师时达72.1±0.4%（闭合76.2%），优于最强匹配计算量标签使用基线（DRIFT+二元奖励）+4.4±0.7点。所有实验仅在一台Apple Mac Studio（M系列，64GB统一内存）上完成，表明无需数据中心级硬件即可实现强推理性能的知识蒸馏。
原文摘要 · Abstract (English)
On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We present CADENCE, a unified framework with a targeted fix for each. Its DRIFT mechanism schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories (per-token surrogates, not sequence-level KL gradient estimators). Six components extend it: (A) COVA, a coverage-adaptive $β$ schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (C) CCD, a dense reward adding numerical-proximity partial credit for incorrect-but-close traces; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-matching calibration regularizer; (F) BSD, a bootstrapped self-distillation phase. On GSM8K and MATH-500 (corrected 512-token protocol, 5 seeds, reported std), CADENCE distills a 0.5B student from a 1.5B teacher to 69.8 $\pm$ 0.5% GSM8K pass@1 (from 48.7% pretrained; 63.2% of the teacher gap closed) and to 72.1 $\pm$ 0.4% with a 3B teacher (76.2% closed), beating the strongest matched-compute label-using baseline (DRIFT+binary reward) by +4.4 $\pm$ 0.7 points. All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。