arXiv:2607.28639cs.CLcs.AI2026-07

小模型知识蒸馏会加剧偏见,需关注拒绝行为变化

The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models

论文配图:The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models
图 1 · 摘自论文原文
  • 区分明确与模糊任务,发现蒸馏对拒绝行为有不同影响
  • 在明确任务中减少偏见,但在模糊任务中导致49%拒绝失效
  • 提出新评估框架PCCD,可识别蒸馏带来的隐藏危害

我们发现,在小型指令微调语言模型中,知识蒸馏对偏见的影响具有不对称性。在明确任务(BBQ-disambig)上,从Mistral-7B教师蒸馏后,最偏倚的基线模型(SmolLM2-1.7B-Instruct)在非回避回答中,错误依从刻板印象的比例从44.5%降至37.2%,准确率从0.55升至0.61。但在模糊任务(BBQ-ambig)上,蒸馏导致拒绝能力下降:15%原本正确回避的案例转为给出刻板答案(沉默损失),且拒绝模式相关性仅弱保留(Spearman rho=0.44)。该损害在另一学生模型(OLMo-2-1B-Instruct)上更严重,沉默损失达49%,新偏见中95%为填充沉默。两个看似更强的结果实为误判:未条件化指标虚高,经条件调整后改善仅从44.5%降至39.8%;跨条件独立性现象在修正解析器后转为正相关(rho=0.58, p<0.01)。聚合指标(CrowS-Pairs、BBQ总体刻板依赖得分)掩盖了个体层面的伤害。本文提出三步诊断框架PCCD,评估拒绝模式保持、承诺回答上下文遵循和能力保留,但本研究网格中无任何配置通过全部三项。

原文摘要 · Abstract (English)

We show that knowledge distillation (KD) in small instruction-tuned language models has asymmetric effects on bias, and that measuring them correctly requires accounting for where refusal mass moves and what the parser can legitimately score. On unambiguous tasks (BBQ-disambig), response-based distillation from a Mistral-7B teacher genuinely improves context-following for the most context-biased baseline (SmolLM2-1.7B-Instruct): among committed (non-abstaining) answers, the rate of overriding correct context with a stereotype falls from 44.5% to 37.2%, with accuracy rising from 0.55 to 0.61. On ambiguous tasks (BBQ-ambig), the same distillation degrades conditional refusal: 15% of the cases where the baseline correctly abstained instead receive stereotype answers (silence-loss), and the distilled refusal pattern only weakly preserves the baseline's (Spearman rho=0.44). The harm reproduces, aggravated, on a second student family (OLMo-2-1B-Instruct): silence-loss reaches 49% and filled-silence accounts for 95% of new bias. Two apparently stronger results are artifacts. An unconditioned override metric reports a 44% -> 23% improvement under a Gemma-2-9B teacher that shrinks to 44.5% -> 39.8% once conditioned on committed answers: the model abstains on 43% of items and its accuracy collapses from 0.55 to 0.35. An apparent cross-condition independence reverses to a positive correlation (rho=0.58, p<0.01) on the valid 19-configuration grid once parser-invalid logit-KD configurations are excluded and the parser is corrected. Aggregate metrics (CrowS-Pairs, overall BBQ Stereotype Reliance Score) average over both effects and conceal the per-item harm. We propose Per-Condition Calibration Diagnosis (PCCD), a three-step protocol evaluating refusal-pattern preservation, committed-answer context-following, and capability preservation. No configuration in our grid passes all three steps.

知识蒸馏模型偏见评估方法小模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。