研究代码模型如何共享编程语言概念，发现英文是核心概念载体。
How Programming Concepts and Neurons Are Shared in Code Language Models
- 用21种语言对测试模型，发现中间层更倾向英文词汇
- 底层存语言特有神经元，顶层出现通用代码概念
- 与多语言对齐的代码语言更接近模型核心概念空间
多项研究探讨了大语言模型在编程任务中的机制，但大多聚焦于单个编程语言（PL）。本文探究多个编程语言与英语在模型概念空间中的关系。我们使用两个基于Llama的模型，在21组编程语言对上执行少样本翻译任务。通过解码中间层嵌入，发现概念空间更接近英语（包括编程语言关键词），且在中间层后半段对英语词元赋予高概率。分析11种编程语言和英语的神经元激活，发现语言特异性神经元主要集中在底层，而每种语言独有的神经元多出现在顶层。与多种语言高度对齐的编程语言难以识别其语言特异性神经元，这类语言通常拥有更大的关键词集，并无论输入输出语言如何，都更接近模型的概念空间。研究揭示了模型内部表示编程语言的结构规律。
原文摘要 · Abstract (English)
Several studies have explored the mechanisms of large language models (LLMs) in coding tasks, but most have focused on programming languages (PLs) in a monolingual setting. In this paper, we investigate the relationship between multiple PLs and English in the concept space of LLMs. We perform a few-shot translation task on 21 PL pairs using two Llama-based models. By decoding the embeddings of intermediate layers during this task, we observe that the concept space is closer to English (including PL keywords) and assigns high probabilities to English tokens in the second half of the intermediate layers. We analyze neuron activations for 11 PLs and English, finding that while language-specific neurons are primarily concentrated in the bottom layers, those exclusive to each PL tend to appear in the top layers. For PLs that are highly aligned with multiple other PLs, identifying language-specific neurons is not feasible. These PLs also tend to have a larger keyword set than other PLs and are closer to the model's concept space regardless of the input/output PL in the translation task. Our findings provide insights into how LLMs internally represent PLs, revealing structural patterns in the model's concept space. Code is available at https://github.com/cisnlp/code-specific-neurons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。