用扩散几何比较神经网络表征,实现跨层跨网络的精准对比。
From Layers to Networks: Comparing Neural Representations via Diffusion Geometry
- 将相似性度量转化为马尔可夫矩阵形式,支持多尺度分析。
- 在14个模型、7个数据集上达到最优准确率与相关性表现。
- 适用于语言与视觉任务,尤其擅长处理分布外数据场景。
扩散几何是一种基于马尔可夫转移矩阵定义的随机游走框架,可从多尺度刻画数据集的流形结构。本文首次将多视图学习工具引入该领域,用于神经表征分析。关键发现是:一类广泛存在的基于表示相似性矩阵(RSM)的相似性度量,在行随机马尔可夫矩阵下具有闭式等价形式,从而可利用扩散几何进行操作。作为首项应用,我们提出中心核对齐(CKA)与距离相关性的多尺度变体,通过使用底层转移矩阵的t次幂,调节扩散尺度以探测数据几何结构。进一步地,我们引入交替扩散方法融合多个层的马尔可夫矩阵,生成单一算子以捕捉网络整体样本几何结构,实现跨层乃至跨网络的相似性计算。我们在包含14种架构、7个数据集、3个领域的表示相似性(ReSi)基准上进行了大量实验,结果在语言与视觉任务中均取得最优精度与输出相关性。此外,在评估分布外数据表现的附加基准上也达到最先进水平。
原文摘要 · Abstract (English)
Diffusion geometry is a manifold learning framework that uses random walks defined by Markov transition matrices to characterize the geometry of a dataset at multiple scales. We use diffusion geometry for neural representations, incorporating tools from multi-view learning into this field for the first time. Our key technical observation is that a broad class of similarity measures based on representational similarity matrices (RSMs) admits a closed-form equivalent formulation in terms of row-stochastic Markov matrices, opening the door to manipulations from diffusion geometry. As a first application, we develop multi-scale variants of Centered Kernel Alignment and Distance Correlation, which utilise the $t^{th}$ power of the underlying transition matrix to probe the data geometry at adjustable diffusion scales. Going further, we introduce variants of these measures which fuse the Markov matrices of several layers via alternating diffusion into a single operator that captures the network's joint sample geometry, allowing similarity to be computed across multiple layers and shifting the comparison from layer-to-layer to network-to-network. We perform extensive numerical experiments, evaluating our measures on the Representational Similarity (ReSi) benchmark comprising 14 architectures trained on 7 datasets across three different domains. Our methods achieve SoTA results in accuracy and output correlation for both language and vision tasks across different models. We furthermore show SoTA performance on an additional benchmark evaluating on out-of-distribution data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。