用新方法发现大模型对概念的结构编码既相似又不同。
Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures
- 提出对比差分CKA，无须训练即可分离概念特异性结构对齐
- 不同架构在概念层面存在中等几何收敛与几乎完美功能迁移
- 适合研究模型可解释性、架构比较与跨模型概念监控
不同大语言模型架构是否以结构兼容的方式编码高层次概念？我们系统分析了一种几何-功能普遍性解耦现象：在多个概念领域和架构家族中，中等几何收敛与近乎完美的功能迁移并存。通过无需训练的对比差分核相关分析（CKA_Delta），该方法基于样本级对比差异计算核对齐，成功分离出概念特异性收敛，实现标准CKA无法达到的显著区分效果。该解耦现象在六个测试概念域中均复现（五个领域几何区分性显著，p ≤ 0.017；安全相关呈趋同功能趋势，p = 0.08），包括两个非指令类概念（代码与自然语言、推理与回忆）且无需系统提示验证。单个70B-70B模型对显示普遍性可能随规模增强，需更多≥70B模型验证。我们定位CKA_Delta为实用的架构分类器与异常检测工具（Gemma: d = 1.08, AUC = 0.79），而非绝对迁移精度预测器，为跨架构概念监控提供无训练诊断手段。
原文摘要 · Abstract (English)
Do different LLM architectures encode high-level concepts in structurally compatible ways? We systematically characterize a geometric-functional universality dissociation: across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Using contrastive-difference CKA (CKA_Delta), a training-free diagnostic that computes kernel alignment on per-sample contrastive differences, we isolate concept-specific convergence from generic similarity -- achieving significant discrimination where standard CKA cannot. The dissociation replicates across all six concept domains we test (five with p <= 0.017 geometric discrimination and safety as a converging-functional trend, p = 0.08), including two non-instruction concepts (code-vs-NL, reasoning-vs-recall) validated without system prompts; a single 70B--70B pair provides an observational note that universality may strengthen with scale, requiring replication with additional >=70B models. We position CKA_Delta as a practical regime classifier and architectural outlier detector (Gemma: d = 1.08, AUC = 0.79) rather than an absolute transfer-accuracy predictor, providing a training-free diagnostic for cross-architecture concept monitoring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。