针对长尾数据集蒸馏偏差问题,提出统一框架实现无偏恢复与软重标注。
Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling
- 从统计对齐视角出发,联合优化模型偏差与监督信号公平性
- 在CIFAR-100-LT上提升15.6%准确率,Tiny-ImageNet-LT上提升11.8%
- 适合关注长尾学习与数据蒸馏的科研人员和工程实践者
数据蒸馏通过生成小规模蒸馏集来高效训练模型,但现有方法在长尾分布下表现不佳,因类别频数不平衡导致模型表征偏差及批归一化(BN)等统计估计被污染。本文重新思考长尾数据蒸馏问题,摒弃轨迹依赖方法,转而采用统计对齐视角,联合缓解模型偏差并恢复公平监督。提出三项核心组件:(1)增强专家模型(观察者模型用于恢复、教师模型用于生成软标签),以实现可靠统计估计与软标签生成;(2)通过动态调整动量的全前向传播重校准BN统计量,减少表示偏斜;(3)基于多轮机制逐步选择高置信度且多样化的增强样本初始化合成图像,提升覆盖度与多样性。在四个长尾基准测试上实验表明,本方法在不同类别失衡程度下均显著优于当前最优方法。特别地,在IPC=10、IF=10条件下,于CIFAR-100-LT上实现15.6%的准确率提升,于Tiny-ImageNet-LT上提升11.8%。
原文摘要 · Abstract (English)
Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance. Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10. Codes are available at https://github.com/2018cx/RLDD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。