arXiv:2604.18135cs.CVcs.AI2026-04TPAMI被引 1

解决大规模数据蒸馏中软标签过大的问题,显著压缩存储并提升精度。

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

论文配图:Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
图 1 · 摘自论文原文
  • 通过分组合成与动态知识重用增强图像和监督多样性。
  • ImageNet-1K标签减少78倍,精度提升7.2%;ImageNet-21K标签减少500倍,精度提升2.8%。
  • 适合需要高效压缩数据集的模型训练场景,尤其适用于大尺度蒸馏任务。

大规模数据蒸馏需存储远超压缩图像体积的辅助软标签:在ImageNet-1K上可达30-40倍,在ImageNet-21K上可达200倍,严重违背压缩目标。我们识别出两个根本原因:(1) 图像多样性不足,合成图像类内相似度过高导致需大量增广;(2) 监督信号多样性不足,训练期间监督信号种类有限,导致高压缩率下性能下降。为此,我们提出针对大规模蒸馏的软标签剪枝与量化方法(LPQLD)。通过类别分批与批归一化监督增强图像多样性;引入动态知识重用的标签剪枝以提升每增广样本的标签多样性,以及基于校准师生对齐的标签量化以提升每图像的增广多样性。实验表明,LPQLD在ImageNet-1K上将软标签存储减少78倍,精度提升7.2%;在ImageNet-21K上减少500倍,精度提升2.8%。跨网络结构与蒸馏方法的广泛实验验证了其优越性。代码已开源。

原文摘要 · Abstract (English)

Large-scale dataset distillation requires storing auxiliary soft labels that can be 30-40x larger on ImageNet-1K and 200x larger on ImageNet-21K than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues necessitating such extensive labels: (1) insufficient image diversity, where high within-class similarity in synthetic images requires extensive augmentation, and (2) insufficient supervision diversity, where limited variety in supervisory signals during training leads to performance degradation at high compression rates. To address these challenges, we propose Label Pruning and Quantization for Large-scale Distillation (LPQLD). We enhance image diversity via class-wise batching and batch-normalization supervision during synthesis. For supervision diversity, we introduce Label Pruning with Dynamic Knowledge Reuse to improve label-per-augmentation diversity, and Label Quantization with Calibrated Student-Teacher Alignment to improve augmentation-per-image diversity. Our approach reduces soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments validate the superiority of LPQLD across different network architectures and dataset distillation methods. Code is available at https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation.

数据蒸馏软标签压缩图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。