用随机答案干扰测试集，防止大模型偷偷刷分。
CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
- 给正确答案加随机干扰，降低理论最高分
- 只要模型得分超过理论上限，就是作弊信号
- 适合公开评测但防数据泄露的场景
发布大型语言模型（LLM）基准测试（尤其是真实答案）到互联网，可能污染未来模型训练并导致评估操纵：模型可能被无意或故意利用这些答案进行训练、选择或针对标签过拟合以篡改排行榜。常见缓解方式是将基准私密化，由参与者提交模型或预测结果，但仍有通过反馈循环实现测试集过拟合的风险。为此，我们提出CapBencher，一种在不完全公开真实答案的前提下发布基准的方法，同时保持对LLM的开放评估。核心思想是通过准备多个逻辑正确的答案，并仅在基准中包含其中一个作为标准答案，从而引入随机性，降低最优可能准确率（即贝叶斯准确率）。该方法既隐藏了真实答案，又能检测泄露或操纵：即使能力最强的模型也无法超过贝叶斯准确率，任何超出者都是强烈警示信号。我们从理论上和实证上证明，CapBencher能有效检测多种基准、模型、训练方法及场景下的测试集过拟合。
原文摘要 · Abstract (English)
Publishing a large language model (LLM) benchmark (especially its ground-truth answers) on the Internet risks contaminating future LLMs and enabling evaluation gaming: it may be unintentionally (or intentionally) used to train or select a model, or exploited to overfit and hack leaderboards when labels are accessible. A common mitigation is to keep the benchmark private and let participants submit their models or predictions to the organizers, but this still permits test-set overfitting through feedback loops. To overcome this issue, we propose CapBencher, a way to publish benchmarks without fully disclosing the ground-truth answers, while preserving open evaluation of LLMs. The main idea is to reduce the best possible accuracy, i.e., Bayes accuracy, by injecting randomness to the answers by preparing several logically correct answers, and only include one of them as the solution in the benchmark. Not only does this obscure the ground-truth answers, but it also offers a test for leakage or gaming: since even fully capable models should not surpass the Bayes accuracy, any model that does is a strong signal. We show theoretically and empirically that CapBencher accurately detects test-set overfitting across diverse benchmarks, models, training methodologies, and scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。