用双曲空间优化数据蒸馏,提升模型性能与稳定性。
Hyperbolic Dataset Distillation
- 将数据特征嵌入双曲空间,利用其层级结构建模能力
- 仅需20%核心集即可保持性能,训练更稳定
- 首个将双曲空间用于数据蒸馏的工作,适合高效学习场景
为应对大规模数据集带来的计算与存储挑战,数据蒸馏被提出以生成紧凑替代数据集,保持模型性能。传统基于优化的方法需复杂双层优化,而分布匹配(DM)方法通过对齐合成与原始数据分布,避免嵌套优化,提升效率。然而,现有方法受限于欧氏空间,将数据视为独立同分布点,忽视其复杂的几何与层次关系。为此,本文提出新型双曲数据蒸馏方法HDD。双曲空间具有负曲率和随距离指数增长的体积特性,天然适合建模树状层级结构。HDD将浅层网络提取的特征嵌入Lorentz双曲空间,通过计算合成与原始数据中心之间的双曲(测地线)距离来衡量差异,并优化该距离,使合成样本自然聚集于原始数据分布的根中心区域,同时保留其几何特征。此外,发现双曲空间中的剪枝仅需20%的蒸馏核心集即可维持性能,显著提升训练稳定性。据我们所知,这是首个将双曲空间引入数据蒸馏的研究。代码已公开于https://github.com/Guang000/HDD。
原文摘要 · Abstract (English)
To address the computational and storage challenges posed by large-scale datasets in deep learning, dataset distillation has been proposed to synthesize a compact dataset that replaces the original while maintaining comparable model performance. Unlike optimization-based approaches that require costly bi-level optimization, distribution matching (DM) methods improve efficiency by aligning the distributions of synthetic and original data, thereby eliminating nested optimization. DM achieves high computational efficiency and has emerged as a promising solution. However, existing DM methods, constrained to Euclidean space, treat data as independent and identically distributed points, overlooking complex geometric and hierarchical relationships. To overcome this limitation, we propose a novel hyperbolic dataset distillation method, termed HDD. Hyperbolic space, characterized by negative curvature and exponential volume growth with distance, naturally models hierarchical and tree-like structures. HDD embeds features extracted by a shallow network into the Lorentz hyperbolic space, where the discrepancy between synthetic and original data is measured by the hyperbolic (geodesic) distance between their centroids. By optimizing this distance, the hierarchical structure is explicitly integrated into the distillation process, guiding synthetic samples to gravitate towards the root-centric regions of the original data distribution while preserving their underlying geometric characteristics. Furthermore, we find that pruning in hyperbolic space requires only 20% of the distilled core set to retain model performance, while significantly improving training stability. To the best of our knowledge, this is the first work to incorporate the hyperbolic space into the dataset distillation process. The code is available at https://github.com/Guang000/HDD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。