用简单方法消除语言模型的回答偏见，提升评测准确性
RBCorr: Response Bias Correction in Language Models
- 基于对数概率修正模型回答倾向，无需复杂训练
- 在12个模型上验证，显著降低偏见并提升性能
- 适合小模型和闭合问答评测，提升结果可信度
语言模型常存在回答偏见，表现为固定选项问题中的偏好倾向。为此，本文提出一种简单有效的响应偏见校正方法（RBCorr），在12个开源语言模型上，针对是/否、蕴含关系及多选题进行测试。结果表明，校正前模型普遍存在偏见，而RBCorr能有效消除偏见并提升性能。同时研究发现，基于对数概率的校正效果高度依赖于模型、数据集和提示格式。总体而言，RBCorr是一种易用方法，可显著提升小模型表现，并使闭合问答基准上的评分更贴近模型真实能力。
原文摘要 · Abstract (English)
Language models (LMs) are known to be prone to response biases, which present as option preference biases in fixed-response questions. It is therefore imperative to develop low-cost and effective response bias correction methods to improve LM performance and enable more accurate evaluations of model abilities. Here, we propose a simple response bias correction strategy ($\texttt{RBCorr}$) and test it on 12 open-weight language models using yes-no, entailment, and multiple choice questions. We show that response bias is prevalent in LMs pre-correction and that $\texttt{RBCorr}$ effectively eliminates bias and boosts model performance. We also explore the generalizability of bias behavior across models, datasets, and prompt formats, showing that LogProbs-based correction is highly dependent on all three of these aspects. Overall, $\texttt{RBCorr}$ is an easy-to-use method that can boost the performance of smaller LMs and ensure that LM performance on closed-response benchmarks aligns more closely with their true capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。