提出新评估方法与干预策略,更真实地衡量模型去污染效果。
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

- 按题目难度分组计算解题概率差距,避免误差抵消
- 实测表明旧方法高估修复效果,新策略显著降低差距
- 适合关注评测公正性与模型真实能力的研究者
公开基准测试数据不可避免地泄露至预训练语料中,导致模型记忆后评分虚高。现有去污染评估采用的G-AP(聚合性能差距)指标存在缺陷:离散的对错判断无法刻画每题表现,先平均再差分会使过抑制与欠抑制相互抵消,统一加权鼓励将解题概率推向干净模型高频值。本文提出SA-PPG(分层聚合每题概率差距):通过采样估计每题解题概率,与干净模型逐题对比,再按干净模型解题概率分组聚合。现有策略依赖对污染位置的估计后再操作,准确性受限于估计精度。RailCap则在生成时实时判断污染:一旦样本回退至贪婪路径,下一词元即被限制为次优选项,持续累积抑制直至输出分布充分分散。在多个污染模型与基准上,SA-PPG显示先前策略的修复效果被严重高估,而RailCap实现最低的SA-PPG。
原文摘要 · Abstract (English)
Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. \textbf{Contamination mitigation evaluation} intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the \textbf{G-AP} (\textbf{G}ap of \textbf{A}ggregate \textbf{P}erformance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose \textbf{SA-PPG} (\textbf{S}tratified \textbf{A}ggregate of \textbf{P}er-question \textbf{P}robability \textbf{G}aps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. \textbf{RailCap} instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。