测试代码大模型压缩后是否真学懂了老师,发现学生模型常被攻击击穿。
A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
- 用变形测试法对比师生模型输出,评估深层行为一致性。
- 学生模型在对抗攻击下性能下降最高达285%,传统评测未发现。
- 提出MetaCompress框架,可识别62%的行为差异,适合模型压缩研究者。
基于Transformer的代码语言模型在众多软件分析任务中表现优异,但因计算成本高、推理慢和环境影响大,实际部署受限。为应对这一挑战,近期研究多采用知识蒸馏将大型代码语言模型(教师)压缩为小型模型(学生),同时保持性能。然而,学生模型在预测行为与内部表征上对教师的深度模仿程度尚不明确,现有基于准确率的评估仅提供表面视图,难以捕捉师生模型间更深层次的行为偏差。本文实证表明,学生模型常未能深度模仿教师,导致在对抗攻击下性能下降最高达285%,而传统评估无法察觉此问题。为此,我们提出MetaCompress——一种基于变形测试的系统性评估框架,通过比较师生模型在一系列行为保真型变形关系下的输出,来评估行为一致性。我们在两个主流任务上,使用三种不同知识蒸馏方法(Compressor、AVATAR、MORPH)压缩的流行代码语言模型进行评估。结果表明,MetaCompress可识别出高达62%的行为差异,凸显了在知识蒸馏流程中引入行为一致性评估的必要性,并确立了MetaCompress作为评估代码语言模型压缩效果的实用框架。
原文摘要 · Abstract (English)
Transformer-based language models of code have achieved state-of-the-art performance across a wide range of software analytics tasks, but their practical deployment remains limited due to high computational costs, slow inference speeds, and significant environmental impact. To address these challenges, recent research has increasingly explored knowledge distillation as a method for compressing a large language model of code (the teacher) into a smaller model (the student) while maintaining performance. However, the degree to which a student model deeply mimics the predictive behavior and internal representations of its teacher remains largely unexplored, as current accuracy-based evaluation provides only a surface-level view of model quality and often fails to capture more profound discrepancies in behavioral fidelity between the teacher and student models. To address this gap, we empirically show that the student model often fails to deeply mimic the teacher model, resulting in up to 285% greater performance drop under adversarial attacks, which is not captured by traditional accuracy-based evaluation. Therefore, we propose MetaCompress, a metamorphic testing framework that systematically evaluates behavioral fidelity by comparing the outputs of teacher and student models under a set of behavior-preserving metamorphic relations. We evaluate MetaCompress on two widely studied tasks, using compressed versions of popular language models of code, obtained via three different knowledge distillation techniques: Compressor, AVATAR, and MORPH. The results show that MetaCompress identifies up to 62% behavioral discrepancies in student models, underscoring the need for behavioral fidelity evaluation within the knowledge distillation pipeline and establishing MetaCompress as a practical framework for testing compressed language models of code derived through knowledge distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。