通过聚类代码模型嵌入,发现其内部隐含的语法语义概念。
Analyzing Latent Concepts in Code Language Models
- 用混合标注法结合静态分析与大模型提示,规模化标注潜在概念。
- 发现概念在层间分布稳定,微调后可预测演变,敏感度指数0.288。
- 概念增强解释使人类理解准确率提升37个百分点,适合可信编程应用。
大型代码语言模型的内部行为解释仍是关键挑战,尤其在需要信任、透明和语义鲁棒性的场景中。我们提出代码概念分析(CoCoA):一种全局后验可解释性框架,通过聚类上下文感知的词元嵌入,揭示代码语言模型表示空间中的涌现词汇、语法和语义结构。我们设计了一种混合标注流程,结合静态分析工具的语法对齐与提示工程的大模型,实现跨抽象层级的可扩展概念标注。分析了概念在各层及三个微调任务中的分布,发现涌现概念簇可识别意外的潜在交互,并用于检测模型学习表征中的趋势与偏差。进一步将局部归因方法与概念分析结合,生成基于概念的解释,显著提升词元级显著性的连贯性与可读性。多模型多任务实证评估显示,该方法发现的概念在语义保持扰动下仍具稳定性(平均聚类敏感度指数,CSI = 0.288),且随微调呈现可预测演化。在编程语言分类任务的用户研究中,概念增强解释有效区分词元角色,相比仅使用集成梯度的词元级归因,人类可解释性提升37个百分点。
原文摘要 · Abstract (English)
Interpreting the internal behavior of large language models trained on code remains a critical challenge, particularly for applications demanding trust, transparency, and semantic robustness. We propose Code Concept Analysis (CoCoA): a global post-hoc interpretability framework that uncovers emergent lexical, syntactic, and semantic structures in a code language model's representation space by clustering contextualized token embeddings into human-interpretable concept groups. We propose a hybrid annotation pipeline that combines static analysis tool-based syntactic alignment with prompt-engineered large language models (LLMs), enabling scalable labeling of latent concepts across abstraction levels. We analyse the distribution of concepts across layers and across three finetuning tasks. Emergent concept clusters can help identify unexpected latent interactions and be used to identify trends and biases within the model's learned representations. We further integrate LCA with local attribution methods to produce concept-grounded explanations, improving the coherence and interpretability of token-level saliency. Empirical evaluations across multiple models and tasks show that LCA discovers concepts that remain stable under semantic-preserving perturbations (average Cluster Sensitivity Index, CSI = 0.288) and evolve predictably with fine-tuning. In a user study on the programming-language classification task, concept-augmented explanations disambiguated token roles and improved human-centric explainability by 37 percentage points compared with token-level attributions using Integrated Gradients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。