提出表征全同性,量化特征在输入空间中的路径依赖变化。
Gauge-invariant representation holonomy
- 通过全局白化与旋转对齐,定义不变于正交变换的几何统计量。
- 实证显示其随环路半径增大而上升,能区分相似模型的鲁棒性差异。
- 适合研究模型泛化性、训练动态及对抗鲁棒性的研究人员使用。
深度网络学习的内部表征具有几何结构——特征如何弯曲、旋转和演化——这影响模型的泛化与鲁棒性。现有相似度度量如CKA或SVCCA仅捕捉激活集的逐点重叠,忽略了表征沿输入路径的变化。两个模型在这些度量下可能几乎相同,但在扰动或对抗攻击下表现迥异。本文提出表征全同性(representation holonomy),一种规范不变的统计量,用于衡量这种路径依赖性。概念上,全同性量化当特征在输入空间的小环路上进行平行传输时累积的‘扭转’:平坦表征的全同性为零,非零值则揭示隐藏曲率。我们的估计器通过全局白化固定规范,利用共享子空间对齐邻域,并仅用旋转进行Procrustes匹配,再将结果嵌回完整特征空间。我们证明该统计量在正交(以及白化后仿射)变换下不变,建立仿射层的线性零空间,并证明小半径下全同性趋于零。实验表明,全同性随环路半径增大而增加,可区分看似相似的模型,且与对抗攻击和数据破坏的鲁棒性相关。它还能追踪训练过程中特征的形成与稳定过程。这些结果表明,表征全同性是超越逐点相似性的实用且可扩展的诊断工具。
原文摘要 · Abstract (English)
Deep networks learn internal representations whose geometry--how features bend, rotate, and evolve--affects both generalization and robustness. Existing similarity measures such as CKA or SVCCA capture pointwise overlap between activation sets, but miss how representations change along input paths. Two models may appear nearly identical under these metrics yet respond very differently to perturbations or adversarial stress. We introduce representation holonomy, a gauge-invariant statistic that measures this path dependence. Conceptually, holonomy quantifies the "twist" accumulated when features are parallel-transported around a small loop in input space: flat representations yield zero holonomy, while nonzero values reveal hidden curvature. Our estimator fixes gauge through global whitening, aligns neighborhoods using shared subspaces and rotation-only Procrustes, and embeds the result back to the full feature space. We prove invariance to orthogonal (and affine, post-whitening) transformations, establish a linear null for affine layers, and show that holonomy vanishes at small radii. Empirically, holonomy increases with loop radius, separates models that appear similar under CKA, and correlates with adversarial and corruption robustness. It also tracks training dynamics as features form and stabilize. Together, these results position representation holonomy as a practical and scalable diagnostic for probing the geometric structure of learned representations beyond pointwise similarity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。