arXiv:2605.24603cs.CLcs.LG2026-05被引 1

发现代码模型按语法结构而非语义组织神经回路,揭示其内部计算逻辑。

CSP-Atlas: Concept-Specific Neural Circuits in a Sparse Python Transformer

  • 通过控制提示词分解出106个概念的专属神经回路
  • 62.5%的活跃神经元专用于概念,非语义驱动
  • 单行语句构建共享回路,体现计算原子性

一个8层稀疏代码变换器为每个测试的Python构造都发展出专用神经回路,且回路组织遵循清晰的计算原则而非语义类别。通过63,800个受控提示词的边际化分析,提取了106个概念(43种AST节点类型,63个内置对象)的神经回路,并使用对比检查提示词(仅呈现关键词无语法结构)将其分解为概念特异性和词元驱动成分。三大发现:第一,所有106个概念在九个参数设置下均产生非空通用回路,概念特异性排序稳定,排除阈值偏差;第二,AST回路中存在真实概念成分:中后期层最活跃神经元高达62.5%为概念专有,而内置对象回路几乎完全由词元驱动;第三,六类计算原子性构造——Import、ImportFrom、Break、Continue、Pass、Assert——尽管语义无关,却因均为无需嵌套体的单行语句而聚类,这一原子性超簇与基于词元模糊性和结构独特性的四层层级结构表明,模型内部组织追踪计算结构而非意义。方法、完整分解数据与分析代码已公开。

原文摘要 · Abstract (English)

A sparse 8-layer code transformer develops dedicated neural circuitry for every Python construct tested, and that circuitry is organised by a clean computational principle rather than by semantic category. We extract neural circuits for 106 concepts (43 AST node types, 63 builtin objects) by marginalising across 63,800 controlled prompts, and decompose each circuit into concept-specific and token-driven components using contrastive checker prompts that present a keyword token without its associated syntactic structure. Three findings emerge. First, all 106 concepts produce non-empty universal circuits at every one of nine parameter settings, and the ranking of concept-specificity across constructs is stable across the sweep - survival is not an artifact of a permissive threshold. Second, AST circuits contain a genuine concept component distinct from token activation: concept-only neurons constitute up to 62.5% of the loudest-firing neurons at mid-to-late layers, while builtin circuits are almost entirely token-driven. Third, six computationally atomic constructs - Import, ImportFrom, Break, Continue, Pass, Assert - cluster together despite being semantically unrelated, sharing only the property of being single-statement constructs requiring no nested body; this atomicity super-cluster, together with a four-tier hierarchy organised by token ambiguity and structural distinctiveness, shows that the model's internal organisation tracks computational structure rather than meaning. The methodology, full decomposition data, and analysis code are released.

代码生成神经回路稀疏模型计算结构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。