arXiv:2510.27253cs.LGcs.AI2025-10

按数据重要性加权提炼数据集,提升模型性能

Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation

  • 用影响函数评估每条数据重要性,动态分配权重
  • 在多个数据集上实现最高7.8%的准确率提升
  • 可无缝接入现有压缩框架,适合数据质量不均场景

数据集蒸馏将大规模数据集压缩为合成子集,在显著降低存储与计算成本的同时保持接近全量训练的性能。现有方法普遍假设所有真实样本贡献相同,但实际数据中存在信息量高与冗余甚至有害的样本。直接蒸馏全量数据而不考虑质量会损害模型表现。本文提出影响加权蒸馏(IWD),通过影响函数显式建模数据质量,在蒸馏过程中为每个样本分配自适应权重,优先保留有益样本,降低无用或有害样本的影响。由于模块化设计,IWD可无缝集成至多种蒸馏框架。实验表明,引入IWD能提升蒸馏数据质量并增强模型性能,最高达7.8%的准确率增益。

原文摘要 · Abstract (English)

Dataset distillation condenses large datasets into synthetic subsets, achieving performance comparable to training on the full dataset while substantially reducing storage and computation costs. Most existing dataset distillation methods assume that all real instances contribute equally to the process. In practice, real-world datasets contain both informative and redundant or even harmful instances, and directly distilling the full dataset without considering data quality can degrade model performance. In this work, we present Influence-Weighted Distillation IWD, a principled framework that leverages influence functions to explicitly account for data quality in the distillation process. IWD assigns adaptive weights to each instance based on its estimated impact on the distillation objective, prioritizing beneficial data while downweighting less useful or harmful ones. Owing to its modular design, IWD can be seamlessly integrated into diverse dataset distillation frameworks. Our empirical results suggest that integrating IWD tends to improve the quality of distilled datasets and enhance model performance, with accuracy gains of up to 7.8%.

数据蒸馏影响函数模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。