arXiv:2607.16859cs.CV2026-07被引 1

用最终参数效果匹配,让小数据集更高效地复现大模型性能。

Dataset Distillation by Influence Matching

论文配图:Dataset Distillation by Influence Matching
图 1 · 摘自论文原文
  • 通过优化参数影响差异,直接对齐真实数据与合成数据的训练结果。
  • 在Tiny-ImageNet上以每类10样本达到31.5%准确率,比前人高4.7%。
  • 适用于图像分类和视觉语言任务,适合追求高效率数据压缩的研究者。

我们从结果导向视角重新审视数据集蒸馏。不同于对训练过程近似(如梯度或轨迹)的模仿,影响匹配(Inf-Match)直接对齐训练收敛后的最终结果:学习一个紧凑的合成数据集,使其对模型参数的影响与原始全量数据一致。为此,我们提出一种全可微、逐样本的影响估计器,无需耗时的逆海森矩阵计算或凸性假设,仅通过优化动态展开与一阶泰勒近似即可实现线性时间计算。随后,通过最小化合成数据与真实数据间的影响差异,实现结果对齐而非启发式过程模仿。Inf-Match在标准分类基准上取得最优性能。例如,在Tiny-ImageNet上(IPC=10)达31.5%,较NCFM提升4.7%。此外,该方法扩展至视觉语言蒸馏任务,在Flickr30K上以200~1000个合成样本,图像/文本检索平均性能领先,优于NCFM 2.5%。代码将通过https://github.com/hrtan/infmatch发布。

原文摘要 · Abstract (English)

We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5\%, a +4.7\% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5\%. The code will be released via https://github.com/hrtan/infmatch.

数据蒸馏参数影响高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。