提出CODE-DITING,用小模型实现高效高解释性的代码评估。
CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
- 基于推理大模型的知识蒸馏,将能力迁移至小模型
- 1.5B模型性能超越同等规模模型,接近5倍参数大模型
- 7B模型胜过GPT-4o和671B模型,仅用1%参数量
可信的代码片段评估方法在神经代码生成中至关重要。传统方法依赖参考解或可执行测试用例,存在灵活性与可扩展性不足的问题。近期的LLM-as-Judge方法通过直接评估问题描述与生成代码的功能一致性提供了新路径。我们针对三种不同数据集开展全面实证研究,发现基于通用基础模型的方法虽性能好但需复杂提示且缺乏可解释性;而基于推理基础模型的方法虽提示简单、可解释性强,却因参数量巨大导致计算成本高昂。为此,我们提出CODE-DITING,一种在准确性、效率与可解释性间取得平衡的新型代码评估方法。通过构建数据蒸馏框架,我们将DeepSeek-R1671B的推理能力有效迁移到1.5B与7B的CODE-DITING模型中,显著提升评估可解释性并降低计算开销。采用多数投票策略后,CODE-DITING 1.5B在同等参数规模下表现优于所有同类模型,并达到通常需5倍参数才能实现的性能;CODE-DITING 7B超越GPT-4o与DeepSeek-V3 671B,仅使用其1%的参数量。进一步实验表明,CODE-DITING对偏好泄露具有鲁棒性,是代码评估的有力替代方案。
原文摘要 · Abstract (English)
Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose CODE-DITING, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1671B to our CODE-DITING 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, CODE-DITING 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that CODEDITING is robust to preference leakage and can serve as a promising alternative for code evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。