通过重心对齐方法实现神经表示的个体级比较,揭示模型间隐藏的相似性。
Barycentric alignment for instance-level comparison of neural representations
- 引入重心对齐框架,消除神经网络表示中的冗余对称性
- 在单个输入层面发现模型间表示收敛与发散的系统性规律
- 适用于视觉、语言模型及人脑表征的跨个体跨区域对比
跨神经网络比较表示具有挑战性,因表示存在单位任意重排或激活空间旋转等对称性,掩盖了模型间的本质等价。本文提出重心对齐框架,消除这些干扰对称性,构建多个模型共用的通用嵌入空间。不同于以往基于整体刺激集的相似性度量,该框架支持在个体刺激层面定义相似性,揭示出哪些输入引发模型间表示趋同或分化。利用这一个体级相似性,我们识别出预测视觉与语言模型族中表示收敛或发散的系统性输入特征。此外,我们为不同个体和皮层区域的人脑表征构建了通用嵌入空间,实现了人类视觉层级各阶段表示一致性在个体层面的比较。最后,将相同框架应用于纯单模态视觉与语言模型,发现后处理对齐至共享空间后,图像-文本相似度评分与人类跨模态判断高度一致,接近对比训练视觉-语言模型的性能。这强烈表明,独立学习的表示已具备足够几何结构,可实现与人类对齐的跨模态比较。结果表明,以个体刺激为单位的表示相似性分析,能揭示集合层面度量无法检测的现象。
原文摘要 · Abstract (English)
Comparing representations across neural networks is challenging because representations admit symmetries, such as arbitrary reordering of units or rotations of activation space, that obscure underlying equivalence between models. We introduce a barycentric alignment framework that quotients out these nuisance symmetries to construct a universal embedding space across many models. Unlike existing similarity measures, which summarize relationships over entire stimulus sets, this framework enables similarity to be defined at the level of individual stimuli, revealing inputs that elicit convergent versus divergent representations across models. Using this instance-level notion of similarity, we identify systematic input properties that predict representational convergence versus divergence across vision and language model families. We also construct universal embedding spaces for brain representations across individuals and cortical regions, enabling instance-level comparison of representational agreement across stages of the human visual hierarchy. Finally, we apply the same barycentric alignment framework to purely unimodal vision and language models and find that post-hoc alignment into a shared space yields image text similarity scores that closely track human cross-modal judgments and approach the performance of contrastively trained vision-language models. This strikingly suggests that independently learned representations already share sufficient geometric structure for human-aligned cross-modal comparison. Together, these results show that resolving representational similarity at the level of individual stimuli reveals phenomena that cannot be detected by set-level comparison metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。