提出新方法检测大模型评测中的位置偏差,发现只有在特定准确率区间才能测出。
Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks

- 通过穷举答案顺序排列,用卡方检验量化位置偏差
- 仅在60%-95%准确率区间可检测到显著偏差
- 揭示低性能模型受处理负载影响,高性能模型受天花板效应干扰
多项选择题评测中位置偏差被广泛认为是能力比较的混淆因素,但现有测量依赖单次答案顺序打乱,结果混杂了内容噪声与采样随机性。本文提出inspect_permute,一个开源工具,可对每道题进行全部答案顺序排列,报告卡方/克雷默V值及自举置信区间。在五门MMLU学科上,对gpt-4o-mini、claude-haiku-4-5、gemini-2.5-flash、grok-3四个厂商模型进行24,000次温度0生成,预注册虚假预测哈希。结果显示,位置偏差仅在约60%-95%基础准确率的“黄金区域”内可统计检测。低于此区间,处理负载主导;高于此区间,天花板效应压缩方差至卡方检验分辨率以下。可检测的偏差分为两类:单调递减(处理负载,低性能模型)与非单调下降(内容模糊,窄能力带)。标准MMLU将所有前沿模型置于检测带之上,因此无信号应理解为不可测,而非无偏。结合arXiv:2606.26185的天花板效应分析,本工作框定了位置偏差测量的可检区间,使该领域核心问题可验证。代码、数据、预注册均以MIT许可发布。
原文摘要 · Abstract (English)
Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity. I introduce inspect_permute, an open-source extension to the inspect_ai evaluation framework that runs exhaustive answer-order permutations per question and reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals. I apply the tool across four vendors (gpt-4o-mini, claude-haiku-4-5, gemini-2.5-flash, grok-3) on five MMLU subjects, 24,000 API calls under temperature-0 generation, with falsifier predictions pre-registered via a public SHA-256 hash before half the data was observed. Position bias turns out to be statistically detectable only within a roughly 60-95% base-accuracy Goldilocks zone. Below it, processing-load dominance swamps subject-specific signal; above it, ceiling effects compress the variance below the chi-squared test resolution. Detectable cells separate into two mechanism types: monotone A-to-D decrease (processing_load, in low-tier models) and non-monotone D-drop (content_ambiguity, in a narrow capability band). Standard MMLU places every frontier-tier model above the detection band, so absence of signal there should be read as not measurable, not unbiased. Together with the ceiling-effect characterisation in arXiv:2606.26185, this work brackets the detectable region of position-bias measurement and makes the field central question askable in a verifiable form. Package, data, preregistration under MIT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。