探查代码模型如何隐式编码类型信息，发现跨语言类型表示存在且稳定。
TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models
- 用双语代码数据集探测预训练模型残差流中的类型表征。
- 即使在无类型代码中也出现跨语言类型表示，且对语法变化有鲁棒性。
- 适合关注代码模型可解释性与类型语义理解的研究者。
当前最先进的代码模型表现优异，但其内部对类型信息的编码程度仍不明确。本文利用包含Java和Python代码示例的并行数据集，探测预训练代码模型残差流中的内部类型表征。结果表明，即使在无类型代码中，跨语言类型表示依然存在。我们进一步测试隐藏状态是否线性编码由有类型函数调用所隐含的结果类型，通过在一个语言上训练探测器，来推断另一语言中的参数和结果类型。最终发现，这种结构部分具备对词汇扰动和跨语言语法差异的鲁棒性。据我们所知，此前关于代码模型可解释性的研究尚未直接针对形式化类型语义或跨语言类型表示。我们已开源代码与数据集。
原文摘要 · Abstract (English)
State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。