arXiv:2605.18836cs.LGcs.CV2026-05

让精简数据集在分布外场景更可靠,提升泛化能力。

Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation

论文配图:Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation
图 1 · 摘自论文原文
  • 通过谱域梯度分析分离类别判别与领域特有信息。
  • 在多个基准上显著提升分布外性能,且不增加计算开销。
  • 适合追求高效且鲁棒的合成数据研究者使用。

数据蒸馏(DD)旨在生成紧凑的合成数据集,保留原始数据的训练效果。然而,其标准设定假设测试数据与训练数据同分布,这在实际中极少成立。直接对蒸馏数据应用后处理领域泛化(DG)方法效果不佳,因现有DG依赖真实数据的自然多样性,而紧凑的合成数据缺乏这种多样性,且通常伴随大量增强开销,违背数据蒸馏的效率目标。为此,本文提出领域可泛化的数据蒸馏(DGDD),明确针对蒸馏数据的分布外(OOD)泛化能力。基于广泛采用的分布匹配(DM)基线,我们发现DM在分布外场景表现差的原因在于类别判别与领域特异性信息在压缩数据集中纠缠。为此,提出谱梯度手术(SGS),通过分析各领域梯度在谱域中的跨域一致性,识别出共享的判别性成分和领域特异性成分。SGS在标准DM更新基础上引入两个互补梯度:一个强化跨域共享分量,另一个显式促进蒸馏数据内部多样性。在多尺度基准上的大量实验表明,SGS显著提升分布外泛化能力,同时保持与现有DM方法的即插即用兼容性。

原文摘要 · Abstract (English)

Dataset Distillation (DD) synthesizes a compact synthetic dataset that preserves the training utility of a full dataset. However, its standard formulation assumes that test data follow the same distribution as training data, an assumption that rarely holds in practice. A straightforward extension-applying post-hoc Domain Generalization (DG) techniques to distilled data-is ill-suited because existing DG methods rely on the natural diversity of real datasets, which compact synthetic sets inherently lack, while also incurring substantial augmentation overhead that conflicts with the efficiency objective of dataset distillation. To address this limitation, we introduce Domain Generalizable Dataset Distillation (DGDD), a new problem setting that explicitly targets out-of-distribution (OOD) generalization of distilled datasets. We study this problem through a widely adopted DD baseline of Distribution Matching (DM). We attribute the OOD vulnerability of DM to the entanglement of class-discriminative and domain-specific information within the compressed synthetic set, and propose Spectral Gradient Surgery (SGS) to disentangle the two. The key insight of SGS is that cross-domain agreement among domain-wise gradients in the spectral domain reveals which gradient components are shared across source domains-and are therefore class-discriminative-and which are domain-specific. Based on this observation, SGS augments the standard DM update with two complementary gradients: one that reinforces cross-domain shared components and another that explicitly promotes diversity within the distilled dataset. Extensive experiments on diverse-scale benchmarks demonstrate that SGS substantially improves OOD generalization while remaining plug-and-play compatible with existing DM methods.

数据蒸馏领域泛化谱分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。