提出几何感知的数据集蒸馏方法,更好保留数据内在结构。
GeoDM: Geometry-aware Distribution Matching for Dataset Distillation
- 在欧氏、双曲、球面空间的乘积流形上进行分布匹配。
- 在多个基准上超越现有方法,且对不同几何策略均有效。
- 适合关注数据几何结构与高效训练的研究者。
数据集蒸馏旨在合成原始数据的紧凑子集,使模型在该子集上训练可达到与在完整大数据集上训练相当的性能。现有分布匹配方法局限于欧氏空间,仅能捕捉线性结构,忽略真实数据的内在几何特性(如曲率)。然而,高维数据常位于低维流形上,表明数据集蒸馏应使蒸馏数据流形与原始数据流形对齐。本文提出几何感知的分布匹配框架GeoDM,运行于欧氏、双曲和球面流形的笛卡尔积空间中,统一建模平坦、层次化和循环结构。为适应底层数据几何,引入可学习的曲率与权重参数以控制三类几何。同时设计最优传输损失以提升分布保真度。理论分析表明,乘积空间中的几何感知分布匹配比欧氏方法具有更小的泛化误差界。在标准基准上的大量实验表明,本方法优于现有最优数据蒸馏方法,并在单一流形分布匹配策略下仍保持有效性。
原文摘要 · Abstract (English)
Dataset distillation aims to synthesize a compact subset of the original data, enabling models trained on it to achieve performance comparable to those trained on the original large dataset. Existing distribution-matching methods are confined to Euclidean spaces, making them only capture linear structures and overlook the intrinsic geometry of real data, e.g., curvature. However, high-dimensional data often lie on low-dimensional manifolds, suggesting that dataset distillation should have the distilled data manifold aligned with the original data manifold. In this work, we propose a geometry-aware distribution-matching framework, called \textbf{GeoDM}, which operates in the Cartesian product of Euclidean, hyperbolic, and spherical manifolds, with flat, hierarchical, and cyclical structures all captured by a unified representation. To adapt to the underlying data geometry, we introduce learnable curvature and weight parameters for three kinds of geometries. At the same time, we design an optimal transport loss to enhance the distribution fidelity. Our theoretical analysis shows that the geometry-aware distribution matching in a product space yields a smaller generalization error bound than the Euclidean counterparts. Extensive experiments conducted on standard benchmarks demonstrate that our algorithm outperforms state-of-the-art data distillation methods and remains effective across various distribution-matching strategies for the single geometries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。