提出新评估方法，解决长答案偏见问题
Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative
- 用贝叶斯后验概率计算答案得分，显式建模答案长度先验
- 在多个基准上，新方法比标准和归一化方法更少受长度影响
- 无需额外推理，可直接替换现有评分方式，适合模型评估场景
多项选择评估基准通过条件对数似然排名候选答案，但存在长度偏差：因对数似然按词元累加，较长答案通常被低估。常见缓解方法是将得分除以完成长度，但我们实证发现该启发式常过度校正，反而偏向更长答案。本文分析了不同评分规则的适用条件，阐明标准准确率与长度归一化准确率在何种答案长度分布下合适，并揭示其长度偏差机制。基于此，我们提出贝叶斯准确率（Bayesian accuracy），通过显式设定答案长度先验，计算每个候选答案的后验概率，从而消除线性长度效应。该方法是基于似然的多选评估的即插即用替代方案，无需额外前向传播，在多个基准和少样本设置下均表现出更低的实际长度偏差。
原文摘要 · Abstract (English)
Multiple-choice benchmarks that rank candidate completions by conditional log-probability suffer from a length bias: because log-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice. A common mitigation is to normalize scores by completion length, but we show empirically that this heuristic frequently over-corrects, introducing a bias toward longer answers instead. We first analyze these scoring rules, characterizing when standard and length-normalized accuracy are appropriate and how their length biases depend on the distribution of completion lengths. Motivated by this analysis, we introduce \emph{Bayesian accuracy}, a scoring rule that computes the posterior probability of each candidate under an explicit prior over answer length, thereby removing linear length effects. Bayesian accuracy is a drop-in replacement for likelihood-based multiple-choice evaluation, requires no additional forward passes, and consistently exhibits lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。