用大模型教小模型理解跨语言代码相似性,提升检测可靠性。
Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection

- 从大模型提取推理能力,蒸馏到小型开源模型中
- 在多种语言对上提升检测准确率,尤其在分布外数据表现更好
- 通过结构化输出设计,大幅降低推理耗时,适合实际部署
跨语言代码克隆检测(X-CCD)因不同语言编写的语义等价程序表面差异大而困难。尽管大语言模型(LLMs)在语义检测方面表现出色,但其作为黑箱系统存在成本高、可复现性差、隐私风险及输出格式不可靠等问题。特别是小型开源模型难以遵循推理类提示并稳定生成二分类标签。为此,我们提出一种知识蒸馏框架,将DeepSeek-R1的推理能力迁移至紧凑的开源学生模型中用于X-CCD。基于Project CodeNet构建推理导向的合成训练数据,使用LoRA适配器微调Phi3和Qwen-Coder。进一步引入响应稳定化方法,包括强制结论提示、二分类头和对比分类头,并通过预测指标与响应率评估模型行为。在Python–Java、Rust–Java、Rust–Python和Rust–Ruby四组语言对上的实验表明,知识蒸馏显著提升了紧凑模型的可靠性,且在分布外场景下常提升预测性能。此外,分类头变体相比生成式推理大幅减少推理时间。结果表明,结合推理导向蒸馏与响应稳定化,可使小型开源模型更实用、更可靠地应用于X-CCD任务。
原文摘要 · Abstract (English)
Cross-language code clone detection (X-CCD) is challenging because semantically equivalent programs written in different languages often share little surface similarity. Although large language models (LLMs) have shown promise for semantic clone detection, their use as black-box systems raises concerns about cost, reproducibility, privacy, and unreliable output formatting. In particular, compact open-source models often struggle to follow reasoning-oriented prompts and to produce outputs that can be consistently mapped to binary clone labels. To address these limitations, we propose a knowledge distillation framework that transfers reasoning capabilities from DeepSeek-R1 into compact open-source student models for X-CCD. Using cross-language code pairs derived from Project CodeNet, we construct reasoning-oriented synthetic training data and fine-tune Phi3 and Qwen-Coder with LoRA adapters. We further introduce response stabilization methods, including forced conclusion prompting, a binary classification head, and a contrastive classification head, and evaluate model behavior using both predictive metrics and response rate. Experiments on Python--Java, Rust--Java, Rust--Python, and Rust--Ruby show that knowledge distillation consistently improves the reliability of compact models and often improves predictive performance, especially under distribution shift. In addition, classification-head variants substantially reduce inference time compared to generation-based inference. Overall, our results show that reasoning-oriented distillation combined with response stabilization makes compact open-source models more practical and reliable for X-CCD detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。