小模型知识蒸馏中,单种子评估会掩盖崩溃和失败模式,需多种子验证。
Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation
- 用3-6个种子对比8种蒸馏方法,发现种子间性能波动大
- 三类蒸馏法出现双峰崩溃,部分种子准确率低于55%
- 首次发现推理蒸馏的输出截断问题,准确率仅0.9%
函数路由——根据自然语言请求从固定目录中选择正确API调用——是部署难题。小模型学生虽具吸引力,但知识蒸馏效果通常仅以单种子报告,而种子方差在小模型中未知。我们在一个包含740个实例的医疗API路由任务上,使用1.5B的Qwen学生模型与20B教师模型,对比八种KD变体与监督交叉熵,对关键配置采用三到六个种子。结果发现:(i) 单种子标准差在2.8至48.7个百分点之间,覆盖所有低于五点的声称增益;(ii) 七种蒸馏方法中有三种表现出双峰崩溃,至少一个种子在三到五个中准确率低于55%,其余正常训练;第四种显示更高方差;(iii) 崩溃具有不同模式——ce_kd和ce_paraphrase为错误函数选择,reasoning_kd则出现此前未记录的输出截断模式,模型输出推理但未生成函数名(准确率0.9%);(iv) 仅progressive_kd和rank_kd在所有测试种子中避免崩溃,标准差≤3.9个百分点;(v) 朴素跨分割+3.78个百分点增益,在受控的同分割多种子重测中变为-2.70个百分点。因此,单种子评估无法检测小模型知识蒸馏中的核心失败模式。
原文摘要 · Abstract (English)
Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students are attractive but knowledge distillation gains are typically reported single-seed, at scales where seed variance is unknown. On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy, using three to six seeds for key configurations. We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points; (ii) three of seven KD variants exhibit bimodal collapse, with at least one in three to five seeds falling below 55% accuracy while the others train normally, and a fourth showing elevated variance; (iii) collapse has distinct modes -- wrong-function selection for ce_kd and ce_paraphrase, and a previously undocumented output-truncation mode for reasoning_kd, where the model emits reasoning but terminates before producing a function name (0.9% accuracy); (iv) only progressive_kd and rank_kd avoid collapse across observed seeds, with sigma <= 3.9 pp; (v) a naive cross-split +3.78 pp gain from input enrichment reverses to -2.70 pp under controlled within-split multi-seed re-testing. Single-seed evaluation is therefore unable to detect central failure modes in small-model KD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。