arXiv:2410.05103cs.CV2024-10被引 2

让精简数据集在不同模型间通用,提升训练效果。

MetaDD: Boosting Dataset Distillation with Neural Network Architecture-Invariant Generalization

  • 分离数据共性与模型特异性特征,增强泛化能力。
  • 在Tiny-Imagenet上跨架构准确率达30.1%,领先第二名1.7%。
  • 轻量设计可嵌入任意精简方法,适合高效训练场景。

数据集精简(DD)旨在从大规模数据中生成精炼、紧凑的精简数据集,以实现高效训练。当前主要挑战是精简数据对特定神经网络(NN)架构存在依赖性:使用某架构精简的数据,若用于训练另一架构,性能常显著下降。本文提出MetaDD,旨在提升精简数据在不同架构间的泛化能力。具体而言,MetaDD将精简数据分解为元特征(跨架构保持一致的共性特征)和异质特征(每架构独有的特征),并引入架构无关损失函数,实现多架构特征对齐,从而增强元特征、抑制异质特征。作为低内存开销组件,MetaDD可无缝集成至任意现有精简方法。实验表明,其显著提升各类精简方法性能:在50个图像/类(IPC)的精简版Tiny-Imagenet上,跨架构模型准确率最高达30.1%,超越次优方法GLaD 1.7%。

原文摘要 · Abstract (English)

Dataset distillation (DD) entails creating a refined, compact distilled dataset from a large-scale dataset to facilitate efficient training. A significant challenge in DD is the dependency between the distilled dataset and the neural network (NN) architecture used. Training a different NN architecture with a distilled dataset distilled using a specific architecture often results in diminished trainning performance for other architectures. This paper introduces MetaDD, designed to enhance the generalizability of DD across various NN architectures. Specifically, MetaDD partitions distilled data into meta features (i.e., the data's common characteristics that remain consistent across different NN architectures) and heterogeneous features (i.e., the data's unique feature to each NN architecture). Then, MetaDD employs an architecture-invariant loss function for multi-architecture feature alignment, which increases meta features and reduces heterogeneous features in distilled data. As a low-memory consumption component, MetaDD can be seamlessly integrated into any DD methodology. Experimental results demonstrate that MetaDD significantly improves performance across various DD methods. On the Distilled Tiny-Imagenet with Sre2L (50 IPC), MetaDD achieves cross-architecture NN accuracy of up to 30.1\%, surpassing the second-best method (GLaD) by 1.7\%.

数据精简跨架构神经网络泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。