用数据信息矩阵揭示ReLU网络中的数据几何结构
Geometry of Singular Foliations and Learning Manifolds in ReLU Networks via the Data Information Matrix
- 通过数据信息矩阵发现数据的奇异叶状结构
- 奇异点为零测集，多数区域存在正则叶状结构
- 可应用于数据集间距离度量与知识迁移
理解高维空间中真实数据的分布是机器学习诸多任务的关键。本文利用训练好的ReLU分类器，通过数据信息矩阵（DIM）——一种弗希尔信息矩阵的变体——为数据空间赋予自然的几何结构。研究发现，该结构呈现奇异叶状形式，其奇异点位于零测集内，而局部正则叶状结构几乎处处存在。实验表明，数据分布与叶状结构的叶高度相关。此外，通过分析DIM的谱特性，本方法展现出测量不同数据集间距离的潜力，可用于知识迁移任务。
原文摘要 · Abstract (English)
Understanding how real data is distributed in high dimensional spaces is the key to many tasks in machine learning. We want to provide a natural geometric structure on the space of data employing a ReLU neural network trained as a classifier. Through the Data Information Matrix (DIM), a variation of the Fisher information matrix, the model will discern a singular foliation structure on the space of data. We show that the singular points of such foliation are contained in a measure zero set, and that a local regular foliation exists almost everywhere. Experiments show that the data is correlated with leaves of such foliation. Moreover we show the potential of our approach for knowledge transfer by analyzing the spectrum of the DIM to measure distances between datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。