arXiv:2511.17914cs.CVcs.AI2025-11NeurIPS被引 1

解决长尾数据蒸馏中软标签混淆偏差问题,提升小类识别效果。

Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation

  • 提出自适应软标签对齐模块,校正蒸馏过程中的标签偏差。
  • 在ImageNet-1k-LT上,尾部类别准确率提升11.8%,整体达41.4%。
  • 轻量级设计,适配多种蒸馏方法,适用于标签资源有限场景。

数据蒸馏可将大规模数据集压缩为紧凑且信息丰富的合成数据,显著降低存储与训练成本。然而,现有研究多聚焦于平衡数据集,在真实世界长尾分布下表现不佳。本文强调软标签在长尾数据蒸馏中的关键作用,揭示性能下降的内在机制。我们推导了基于蒸馏数据的不平衡感知泛化界,并通过系统扰动数据不均衡程度,识别出两类软标签偏差来源:蒸馏模型和蒸馏图像。为此,提出ADSA(自适应软标签对齐)模块,以校正纠缠偏差。该轻量模块可无缝集成至现有蒸馏流程,持续提升性能。在ImageNet-1k-LT数据集上,使用EDC和IPC=50时,尾部类别准确率最高提升11.8%,整体准确率达到41.4%。大量实验表明,ADSA在标签预算有限及多种蒸馏技术下均具鲁棒性与通用性。代码已开源:https://github.com/j-cyoung/ADSA_DD.git。

原文摘要 · Abstract (English)

Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation. Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels. To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8% and raises overall accuracy to 41.4%. Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques. Code is available at: https://github.com/j-cyoung/ADSA_DD.git.

数据蒸馏长尾学习软标签模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。