代码模型如何理解任务、语言和自身结构的分工？
What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations
- 用电路提取法分离任务、语言、模型三因素的影响
- 任务决定哪些概念被专一表示（相关性ρ≈0.65），模型决定位置与演化方式
- 发现模型间共享神经元模式，如Qwen将多个类型关键字聚成一个簇
我们研究独立训练的代码语言模型是否以相同方式表征内容。通过一个2×2实验设计（Python与Rust交叉Qwen2.5-Coder-7B和DeepSeek-Coder-V1-6.7B），在四个模型中统一测量了58个Python和57个Rust语法概念。结果表明：任务决定哪些概念获得专用电路（Python ρ=0.638，Rust ρ=0.673，p<10⁻⁷）；模型决定电路分布位置（Qwen在第17-19层，DeepSeek在第6-7层）；模型也决定电路演化方式（Qwen早期出现原子概念峰值，而DeepSeek没有）。此外，Rust概念所需电路比Python多2-3倍，两模型共享神经元（6/7与7/7配对构造），且DeepSeek共享更多（1.94倍于Qwen）。Qwen还将Rust类型与特质机制的九个关键词绑定为一个紧凑神经元簇（Jaccard 0.535 vs null 0.112，p<0.001），揭示表面语法之外的语义维度。消融与线性探测验证了电路功能性。所有结论限于该2×2设计，第三模型的预测能力是下一步测试。
原文摘要 · Abstract (English)
Do independently trained language models come to represent the same thing in the same way? We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B -- and measuring a complete inventory of grammatical concepts (58 Python, 57 Rust) identically in all four cells: the smallest design that separates what depends on the task, the language, and the model. The answer splits into three parts. What earns dedicated circuitry is set by the task: the models agree on which concepts receive circuits (Spearman $ρ$ = 0.638 for Python, 0.673 for Rust, both p < $10^{-7}$). Where those circuits sit is set by the model: Qwen processes concepts in a late band (~L17-19), DeepSeek at L6-7, for both languages. How circuits grow across layers is also set by the model: Qwen gives its atomic concepts an early spike that DeepSeek does not. "Are circuits universal?" thus has no single answer: yes for What, no for Where and How -- universality is a property of representational content, not of computational organisation. None of this structure was fixed in advance. The agreement could have landed anywhere between independence and identity; it lands at $ρ\approx 0.65$. Rust constructs receive 2-3x more concept-specific circuitry than their Python equivalents, in both models. Both models share neurons between the languages (6/7 and 7/7 paired constructs), DeepSeek 1.94x more than Qwen -- a direction no prior result predicts. And Qwen binds nine keywords of Rust's type-and-trait machinery into one tight neuron cluster (Jaccard 0.535 vs null 0.112, p < 0.001), a semantic dimension invisible in surface syntax. Ablation and linear probes confirm the circuits are functional. All claims are scoped to this 2x2; whether the per-model profile predicts a third model is the designed next test.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。