arXiv:2412.16895cs.CV2024-12AAAI被引 16

提出自适应数据量化方法,提升小规模数据集的训练效果与泛化能力。

Adaptive Dataset Quantization

  • 基于代表性、多样性与重要性评分实现自适应采样
  • 在多个数据集上比传统方法平均提升3%准确率
  • 适用于资源受限场景,对不同模型架构兼容性好

当前深度学习依赖在大规模数据集上训练复杂神经网络,面临严重的计算挑战。为缓解硬件资源有限带来的存储压力,数据压缩方法如数据蒸馏(DD)和核心集选择(coreset selection)被提出,通过合成或筛选生成紧凑但信息丰富的数据集以实现高效训练。然而,DD需昂贵优化且跨模型泛化能力差,coreset选择则因保留率低且依赖启发式策略,实用性受限。为此,本文提出一种新型通用数据压缩框架——自适应数据量化(ADQ)。首先指出朴素数据量化(DQ)因均匀采样忽略各生成区间的重要性而表现不佳;随后提出新自适应采样策略,通过纹理水平与基于对比学习的多样性评分评估生成区间的代表性、多样性和重要性。大量实验表明,该方法不仅在不同模型架构间具备优异泛化能力,且在多个数据集上超越传统方法平均3%的准确率,达到当前最优水平。

原文摘要 · Abstract (English)

Contemporary deep learning, characterized by the training of cumbersome neural networks on massive datasets, confronts substantial computational hurdles. To alleviate heavy data storage burdens on limited hardware resources, numerous dataset compression methods such as dataset distillation (DD) and coreset selection have emerged to obtain a compact but informative dataset through synthesis or selection for efficient training. However, DD involves an expensive optimization procedure and exhibits limited generalization across unseen architectures, while coreset selection is limited by its low data keep ratio and reliance on heuristics, hindering its practicality and feasibility. To address these limitations, we introduce a newly versatile framework for dataset compression, namely Adaptive Dataset Quantization (ADQ). Specifically, we first identify the sub-optimal performance of naive Dataset Quantization (DQ), which relies on uniform sampling and overlooks the varying importance of each generated bin. Subsequently, we propose a novel adaptive sampling strategy through the evaluation of generated bins' representativeness score, diversity score and importance score, where the former two scores are quantified by the texture level and contrastive learning-based techniques, respectively. Extensive experiments demonstrate that our method not only exhibits superior generalization capability across different architectures, but also attains state-of-the-art results, surpassing DQ by average 3\% on various datasets.

数据压缩自适应采样模型泛化量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。