同一问题不同表述，模型表现可能大逆转。
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
- 设计四类语义不变的重述方法，量化模型对表述的敏感度
- 强模型比弱模型更易因表述变化而失分，且失分远大于得分
- 模型对某些重述特别脆弱，说明问题表述本身是关键瓶颈
基准测试分数源于问题的单一表述，但同一问题可有多种问法。我们发现，在保持语义和答案不变的前提下，重述问题常导致模型答案在正反方向频繁翻转，即原本错误变正确，或正确变错误，这种现象称为‘漂移’。BenchDrift 沿语言、指称、语用和结构四轴生成意义保持的变体，测量各类重述下正确性翻转的频率与原因。在八种模型与三个基准（GSM8K、MMLU、MATH-Hard）上，观察到双向漂移显著。首先，模型越强，对表述越敏感：弱模型重述后得分增益大于损失，而强模型损失远超收益；因此，当前最佳模型的表现最依赖偶然给定的表述。其次，尽管各模型漂移程度不同，但它们对哪些重述会引发错误高度一致，表明脆弱性源自表述而非模型。此外，无论问题变短或变长，重述都会破坏模型原有高置信度的答案。代码与数据：https://github.com/IBM/BenchDrift/tree/demo-ui
原文摘要 · Abstract (English)
A benchmark score comes from a single phrasing of each problem. That single phrasing is treated as if it stood for the whole space of ways the same problem could be asked, but it does not. We show that rephrasing a problem while keeping its meaning and answer fixed routinely flips a model's answer in both directions, so some failures become successes and some successes become failures. We call this drift. BenchDrift generates meaning-preserving variations of benchmark problems along four axes, namely linguistic, referential, pragmatic, and structural, and measures how often, and why, correctness flips under each. Across eight models and three benchmarks (GSM8K, MMLU, MATH-Hard), we observe that drift is large in both directions. Two findings stand out. First, phrasing sensitivity does not fade as models get better. Instead, it changes sign. Weak models gain more from rephrasing than they lose, while strong models lose far more than they gain. We find that the best models on a benchmark are therefore the ones whose scores depend most on the wording they happened to be given. Second, the models largely agree on which rephrasings cost the most correct answers even though they differ in how much they drift, so fragility belongs to the rephrasing and not to the model. Furthermore, rephrasing breaks answers a model was confident about, whether the problem is made shorter or longer. Code and Data: https://github.com/IBM/BenchDrift/tree/demo-ui
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。