arXiv:2411.08687cs.LG2024-11被引 2

用多尺度拓扑谱诊断神经网络收敛,区分局部噪声与全局结构变化。

Diagnosing Neural Convergence with Topological Alignment Spectra

  • 提出拓扑对齐谱(TAS),通过不同邻域大小的归一化交集相似度检测多尺度结构变化。
  • 实验显示微调引发跨尺度的拓扑重组,而随机种子间局部差异大但语义簇对齐主导。
  • 适合研究模型收敛性、表征稳定性及深层网络中结构演化机制的科研人员。

神经网络表征相似性具有固有的尺度依赖性,而广泛使用的中心核对齐(CKA)和Procrustes分析仅提供全局标量估计,常无法区分局部几何抖动与宏观语义重组,将多尺度结构关系压缩为单一无信息值。本文提出拓扑对齐谱(TAS),通过在不同邻域大小下扫查归一化平均杰卡德相似度,构建多尺度诊断工具。通过对齐期望范围(从随机重叠到完全一致)进行归一化,TAS生成维度不变的尺度谱:1表示完全结构对齐,0表示随机水平一致,负值则表明特定尺度上存在主动反对齐。合成点云实验表明,TAS可识别不同类型的对齐扰动:局部抖动破坏细粒度邻域但保留聚类结构,而聚类中心重排则保持局部相似性但破坏全局对齐——这些现象在单标量指标下难以分辨或被混淆。应用于MultiBERTs数据集发现,微调导致跨尺度的全面拓扑重组,挑战了任务适配仅为保守或局部调整的观点。尽管不同随机种子的模型在局部存在差异,语义聚类仍是主导对齐尺度。TAS为深度网络的收敛与表征稳定性提供了细粒度、拓扑感知的诊断替代方案。

原文摘要 · Abstract (English)

Representational similarity in neural networks is inherently scale-dependent, yet widely used metrics such as Centered Kernel Alignment (CKA) and Procrustes analysis provide only global scalar estimates. These scalars often fail to distinguish micro-scale geometric jitter (local noise) from macro-scale semantic reorganization, compressing multi-scale structural relationships into a single uninformative value. We introduce the Topological Alignment Spectrum (TAS), a multi-scale diagnostic tool that sweeps normalized mean Jaccard similarity over varying neighborhood sizes. By normalizing the metric over an analytically-derived expected range (from expected overlap under randomness to perfect alignment), TAS yields a dimension-invariant metric over a spectrum of scales, where one indicates perfect structural alignment, zero reflects chance-level agreement, and negative values signal active anti-alignment at specific scales. Experiments on synthetic point clouds demonstrate that TAS allows the recognition of distinct types of alignment perturbation: local jitter harms fine-grained neighborhoods but preserves cluster-level structure, while cluster-center shuffling preserves local similarity but disrupts global alignment -- phenomena that remain invisible or conflated under global, single-scalar metrics. Applying TAS to the MultiBERTs collection reveals that fine-tuning induces comprehensive topological reorganization across scales, challenging the view of task adaptation as merely conservative or localized. While models from different random seeds remain locally divergent, semantic clusters emerge as the dominant scale of alignment. TAS thus offers a granular, topology-aware alternative for diagnosing convergence and representational stability in deep networks.

神经网络表征分析拓扑学习收敛诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。