arXiv:2501.07575cs.CVcs.AI2025-01被引 13

用多模型投票生成更高质量的精简数据集,提升模型泛化能力。

Dataset Distillation via Committee Voting

  • 通过多个模型投票生成软标签,融合集体知识提升数据代表性。
  • 在多种数据集和每类样本数(IPC)下均优于现有方法。
  • 适合需要高效训练且关注泛化性能的研究者使用。

数据提炼旨在合成一个紧凑但具有代表性的数据集,以实现高效的模型训练。现有方法主要关注提升数据合成与真实数据的对齐性,或扩展到大规模数据集的提炼。本文提出委员会投票数据提炼(CV-DD),一种正交的方法,利用多个模型的集体知识生成更高品质的提炼数据。我们首先建立了一个强基线,通过现代架构和优化选择达到最先进性能。通过整合多个模型的分布与预测并生成高质量软标签,该方法捕捉更广泛的数据特征,减少模型特异性偏差和分布偏移的影响,显著提升泛化能力。基于投票的策略增强了多样性与鲁棒性,缓解过拟合,提升后评估表现。在多个数据集和不同每类样本数(IPC)设置下的大量实验表明,CV-DD始终优于单模型与多模型提炼方法,并能良好泛化至非训练型框架及具有挑战性的合成到真实迁移任务。代码已开源:https://github.com/Jiacheng8/CV-DD。

原文摘要 · Abstract (English)

Dataset distillation aims to synthesize a compact yet representative dataset that preserves the essential characteristics of the original data for efficient model training. Existing methods mainly focus on improving data-synthetic alignment or scaling distillation to large datasets. In this work, we propose $\textbf{C}$ommittee $\textbf{V}$oting for $\textbf{D}$ataset $\textbf{D}$istillation ($\textbf{CV-DD}$), an orthogonal approach that leverages the collective knowledge of multiple models to produce higher-quality distilled data. We first establish a strong baseline that achieves state-of-the-art performance through modern architectural and optimization choices. By integrating distributions and predictions from multiple models and generating high-quality soft labels, our method captures a broader range of data characteristics, reduces model-specific bias and the impact of distribution shifts, and significantly improves generalization. This voting-based strategy enhances diversity and robustness, alleviates overfitting, and improves post-evaluation performance. Extensive experiments across multiple datasets and IPC settings demonstrate that CV-DD consistently outperforms single- and multi-model distillation methods and generalizes well to non-training-based frameworks and challenging synthetic-to-real transfer tasks. Code is available at: https://github.com/Jiacheng8/CV-DD.

数据提炼多模型投票软标签泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。