动态变换代码评测集，防止模型作弊，让评估更真实。
Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models
- 对代码输入进行语义不变的动态变异，生成新测试集。
- 所有模型性能均明显下降，部分排名大幅变化。
- 有效抵抗训练数据泄露问题，适合严谨评估代码模型。
本文针对模型评估中的关键挑战：当代码模型在训练中已接触过评测集时，如何保持评测的有效性？我们提出一种动态基准框架，针对代码理解或推理评测集，通过多种语义保持的变异方式动态转换每个输入（即程序），构建语法新颖但语义相同的全新评测集。我们在10个主流语言模型上进行了评估，结果揭示了几个有趣甚至出人意料的发现：(1) 所有模型表现均显著低于原评测结果；(2) 部分模型间的排名发生剧烈变化；(3) 动态评测集能有效抵御数据污染问题。
原文摘要 · Abstract (English)
In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated 10 popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) dynamic benchmarks can resist against the data contamination problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。