用对抗辩论提升问答评估难度,防止模型作弊式记忆。
Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks
- 将标准问答转为双模型辩论,一辩正确答案,一辩提出新答案。
- 同一模型在测试集微调后准确率从50%升至82%,但辩论表现下降。
- 只需少量裁判模型即可有效区分模型真实推理能力,成本更低。
随着前沿语言模型逐渐饱和于标准问答基准,数据污染、记忆化和高昂的数据集构建成本问题日益突出。我们提出一种辩论驱动的评估范式,将任意现有QA数据集转化为结构化对抗辩论:一个模型负责捍卫官方答案,另一个构建并辩护替代答案,由对正确答案未知的裁判模型进行评判。通过强制多轮论辩,该方法显著提升任务难度,抑制浅层记忆,同时复用原有问答项以降低标注开销。主要贡献包括:(1)系统性转换QA任务为辩论评估的流水线;(2)发布一个公开基准,在MMLU-Pro子集上验证该范式的有效性,包含标准化协议与参考模型。实证结果表明该方法具备鲁棒性且能有效应对数据污染——一个在测试题上微调的Llama 3.1模型准确率从50%提升至82%,但在辩论中表现更差。结果还显示,即使较弱的裁判也能可靠区分更强的辩手,说明该评估方式可扩展至未来更强大模型,同时仅需原基准创建成本的一小部分。整体而言,本框架表明‘在测试集上预训练已不再足够’,为衡量先进语言模型的真实推理能力提供可持续路径。
原文摘要 · Abstract (English)
As frontier language models increasingly saturate standard QA benchmarks, concerns about data contamination, memorization, and escalating dataset creation costs persist. We propose a debate-driven evaluation paradigm that transforms any existing QA dataset into structured adversarial debates--where one model is given the official answer to defend, and another constructs and defends an alternative answer--adjudicated by a judge model blind to the correct solution. By forcing multi-round argumentation, this approach substantially increases difficulty while penalizing shallow memorization, yet reuses QA items to reduce curation overhead. We make two main contributions: (1) an evaluation pipeline to systematically convert QA tasks into debate-based assessments, and (2) a public benchmark that demonstrates our paradigm's effectiveness on a subset of MMLU-Pro questions, complete with standardized protocols and reference models. Empirical results validate the robustness of the method and its effectiveness against data contamination--a Llama 3.1 model fine-tuned on test questions showed dramatic accuracy improvements (50% -> 82%) but performed worse in debates. Results also show that even weaker judges can reliably differentiate stronger debaters, highlighting how debate-based evaluation can scale to future, more capable systems while maintaining a fraction of the cost of creating new benchmarks. Overall, our framework underscores that "pretraining on the test set is no longer all you need," offering a sustainable path for measuring the genuine reasoning ability of advanced language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。