用双重沃尔什距离构建数据集动态流,实现标签数据的高效迁移与压缩。
Flowing Datasets with Wasserstein over Wasserstein Gradient Flows
- 将标签数据视为概率分布的混合,用双重沃尔什距离定义其空间结构
- 设计可计算的梯度流,使数据集在目标函数下降中演化
- 适用于数据迁移与蒸馏,尤其适合处理复杂分布间的对齐
机器学习中许多应用涉及以概率分布形式表示的数据。这类无限维对象需要全新的可计算梯度流技术。例如,对带标签数据集进行流动是领域自适应、迁移学习和数据蒸馏的核心任务。本文将每个类别表示为特征的条件分布,将整个数据集建模为这些类别分布(本身是概率分布)上的混合分布,即把带标签数据集看作概率分布之上的概率分布。我们在此空间上引入最优传输结构,采用双重沃尔什(Wasserstein over Wasserstein, WoW)距离,并推导出该空间上的微分结构,定义了WoW梯度流。该框架可设计减少目标泛函的动力学过程。我们在迁移学习和数据蒸馏任务中应用此方法,结合新的可计算泛函——基于切片沃尔什核的极大均值差异(Maximum Mean Discrepancy),实现高效的数据集演化与优化。
原文摘要 · Abstract (English)
Many applications in machine learning involve data represented as probability distributions. The emergence of such data requires radically novel techniques to design tractable gradient flows on probability distributions over this type of (infinite-dimensional) objects. For instance, being able to flow labeled datasets is a core task for applications ranging from domain adaptation to transfer learning or dataset distillation. In this setting, we propose to represent each class by the associated conditional distribution of features, and to model the dataset as a mixture distribution supported on these classes (which are themselves probability distributions), meaning that labeled datasets can be seen as probability distributions over probability distributions. We endow this space with a metric structure from optimal transport, namely the Wasserstein over Wasserstein (WoW) distance, derive a differential structure on this space, and define WoW gradient flows. The latter enables to design dynamics over this space that decrease a given objective functional. We apply our framework to transfer learning and dataset distillation tasks, leveraging our gradient flow construction as well as novel tractable functionals that take the form of Maximum Mean Discrepancies with Sliced-Wasserstein based kernels between probability distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。