揭示代码大模型内部神经元如何区分语言与共性语义
Neuron-Guided Interpretation of Code LLMs: Where, Why, and How?
- 通过分析神经元响应，定位语言专属与通用语义层
- 低层编码语法特征，中层形成跨语言共享的抽象表示
- 可提升多语言代码生成、克隆检测与摘要任务性能
代码大模型在代码智能任务中表现优异，但其内部可解释性尚未充分探索。现有自然语言处理中的神经元可解释性方法对源代码不适用，因编程语言具有形式化、层级化和可执行性特征。我们针对 Llama-3.1-8B 与 Qwen2.5-Coder-32B 在 C++、Java、Python、Go 及 JavaScript 多语言输入下的代码生成过程，从神经元层面进行实证研究，识别出对特定语言敏感的神经元（语言专属神经元）以及编码语言无关代码表示的前馈层（概念层）。结果表明：（1）存在专用于单个语言的神经元，同时也有支持通用生成的通用神经元子集；（2）底层主要编码语言特异性语法，中层则捕获跨语言共享的语义抽象，构成概念层。我们在三个任务中验证其价值：基于神经元引导的微调提升代码生成效果，利用概念层嵌入实现克隆检测，以及以概念层引导迁移提升代码摘要性能，均在多语言场景下取得一致增益。
原文摘要 · Abstract (English)
Code language models excel on code intelligence tasks, yet their internal interpretability is underexplored. Existing neuron interpretability techniques from NLP are suboptimal for source code due to programming languages formal, hierarchical, and executable nature. We empirically investigate code LLMs at the neuron level, localizing language-specific neurons (selectively responsive to one language) and concept layers (feed-forward layers encoding language-agnostic code representations). We analyze Llama-3.1-8B and Qwen2.5-Coder-32B on multilingual inputs in C++, Java, Python, Go, and JavaScript, measuring neuron selectivity and layerwise contributions during generation. We find (1) neurons specialized for individual languages alongside a universal subset supporting general-purpose generation; and (2) lower layers mainly encode language-specific syntax, while middle layers capture semantic abstractions shared across languages, emerging as concept layers. We demonstrate utility on three tasks: neuron-guided fine-tuning for code generation, clone detection via concept-layer embeddings, and concept-layer-guided transfer for code summarization, each yielding consistent gains in multilingual settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。