arXiv:2607.19956cs.CLcs.AI2026-07

提出可靠性感知蒸馏方法,显著提升低资源语言摘要效果。

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

论文配图:When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
图 1 · 摘自论文原文
  • 根据验证损失方向判断每样本蒸馏有效性,动态过滤有害样本。
  • 在巴格拉语摘要任务上,性能比标准蒸馏提升0.0219 ROUGE-L。
  • 仅用6000万参数,超越50倍大的30亿参数大模型。

知识蒸馏(KD)是压缩序列到序列模型的常用方法,但其对单个样本的影响很少被研究。在BanSum巴格拉语摘要基准上,我们发现标准KD相比交叉熵基线仅提升0.0003 ROUGE-L,且约51.3%的训练样本在标准KD下会增加学生模型的验证损失。为此,我们提出两种互补的可靠性感知蒸馏方法:CHAD通过梯度对齐评估每样本蒸馏效用,并训练轻量门控网络推广此判断;EWAD+CPDP结合词元级熵加权自适应蒸馏与来自另一不兼容词表教师的容量比例几何约束。在BanSum上,两者分别提升0.0173和0.0219 ROUGE-L,而标准KD仅提升0.0003;尽管仅使用6000万参数,两者均优于微调后的Qwen 2.5-3B(50倍更大)。我们进一步在15种语系多样的XL-Sum语言中评估更强的方法EWAD+CPDP,其中在10/15种语言上优于纯交叉熵基线;增益最稳定时两教师提供互补信号,最弱时二者均覆盖不足或饱和。

原文摘要 · Abstract (English)

Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50x larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.

知识蒸馏低资源语言摘要生成模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。