动态生成代码模型测试题，防止数据污染导致的评估失真。
Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- 用多个智能体对原始题目改写，保持逻辑不变但形式多样。
- 在21个代码大模型上验证，有效识别受污染数据下的真实推理能力。
- 适合关注模型评测可信度的研究者与开发者使用。
代码大语言模型的快速演进凸显了对其推理能力进行有效且透明评估的重要性。然而，当前评估方法严重依赖公开的人工创建数据集，这些固定基准数据集易受数据污染影响，这是训练过程中大规模数据收集不可避免的结果。现有缓解数据污染的方法往往受限于人力成本且问题难度分布不均。为此，我们提出 ool，一个针对代码大语言模型在潜在数据污染下评估的新型基准测试套件。给定一个编程问题种子， ool 使用多个智能体在不改变核心逻辑的前提下提取并修改上下文，生成语义等价的变体。我们引入动态数据生成方法，并在两个种子数据集上对21个代码大语言模型进行了实证研究。结果表明， ool 能在数据污染风险下有效评估推理能力，同时生成多样化的问题集，确保评估的一致性与可靠性。
原文摘要 · Abstract (English)
The rapid evolution of code largelanguage models underscores the need for effective and transparent benchmarking of their reasoning capabilities. However, the current benchmarking approach heavily depends on publicly available, human-created datasets. The widespread use of these fixed benchmark datasets makes the benchmarking process to be static and thus particularly susceptible to data contamination, an unavoidable consequence of the extensive data collection processes used to train Code LLMs. Existing approaches that address data contamination often suffer from human effort limitations and imbalanced problem complexity. To tackle these challenges, we propose \tool, a novel benchmarking suite for evaluating Code LLMs under potential data contamination. Given a seed programming problem, \tool employs multiple agents to extract and modify the context without altering the core logic, generating semantically equivalent variations. We introduce a dynamic data generation methods and conduct empirical studies on two seed datasets across 21 Code LLMs. Results show that \tool effectively benchmarks reasoning capabilities under contamination risks while generating diverse problem sets to ensure consistent and reliable evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。