通过细粒度关系蒸馏，让小模型学得更准更快
Local Dense Logit Relations for Enhanced Knowledge Distillation
- 递归拆解重组logit，捕捉类别间精细关系
- 动态调整关键类别对权重，提升学习效率
- 适合追求高精度的小模型训练场景
当前先进的logit蒸馏方法虽具通用性、简洁性和高效性，但对logit知识中的细粒度关系挖掘仍不充分。本文提出局部密集关系logit蒸馏（LDRLD），通过递归解耦与重组logit信息，捕获类别间的复杂关系，为学生模型提供更清晰的指导。引入自适应衰减权重（ADW）策略，结合逆序权重（IRW）与指数排名衰减（ERD），动态调节关键类别对的权重：IRW按类别对排名差值反比赋权，ERD根据类别对总排名得分自适应控制衰减速率。此外，在递归解耦后，蒸馏剩余非目标知识以保证知识完整性并进一步提升性能。在CIFAR-100、ImageNet-1K和Tiny-ImageNet上的大量实验表明，该方法优于现有主流logit蒸馏方法。代码将公开。
原文摘要 · Abstract (English)
State-of-the-art logit distillation methods exhibit versatility, simplicity, and efficiency. Despite the advances, existing studies have yet to delve thoroughly into fine-grained relationships within logit knowledge. In this paper, we propose Local Dense Relational Logit Distillation (LDRLD), a novel method that captures inter-class relationships through recursively decoupling and recombining logit information, thereby providing more detailed and clearer insights for student learning. To further optimize the performance, we introduce an Adaptive Decay Weight (ADW) strategy, which can dynamically adjust the weights for critical category pairs using Inverse Rank Weighting (IRW) and Exponential Rank Decay (ERD). Specifically, IRW assigns weights inversely proportional to the rank differences between pairs, while ERD adaptively controls weight decay based on total ranking scores of category pairs. Furthermore, after the recursive decoupling, we distill the remaining non-target knowledge to ensure knowledge completeness and enhance performance. Ultimately, our method improves the student's performance by transferring fine-grained knowledge and emphasizing the most critical relationships. Extensive experiments on datasets such as CIFAR-100, ImageNet-1K, and Tiny-ImageNet demonstrate that our method compares favorably with state-of-the-art logit-based distillation approaches. The code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。