arXiv:2506.03923cs.CL2025-06被引 5

语言模型在比较题中易被'更多''更少'等词引导,产生系统性偏差。

More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning

  • 设计数学对比基准MathComp,测试不同表述对模型判断的影响
  • 300个题目中,92%的错误与提示词方向一致,体现显著语义偏见
  • 含身份标签的提示会加剧偏差,适合关注公平性的研究者参考

大型语言模型对输入表述敏感,但其语义线索如何影响推理机制尚不明确。本文在具有客观真值的数学比较问题中研究这一现象,发现存在持续且定向的表述偏差:逻辑等价的问题若包含‘更多’‘更少’或‘相等’等词,会系统性地引导模型预测向该词汇方向偏移。为此,我们构建了由300个对比场景组成的可控基准MathComp,覆盖三个大模型家族的14种提示变体。结果表明,模型错误常反映语言引导,多数情况下系统性偏向提示中的比较词。链式思考提示可减轻此类偏差,但效果因格式而异:自由形式推理更具鲁棒性,结构化格式可能保留甚至重引入方向漂移。此外,即使底层数量相同,加入‘女性’‘黑人’等身份术语会放大方向性偏差,揭示语义表述与社会指涉的交互作用。这些发现暴露了标准评估的盲点,推动构建注重表述感知的基准,以诊断大模型推理的鲁棒性与公平性。

原文摘要 · Abstract (English)

Large language models (LLMs) are known to be sensitive to input phrasing, but the mechanisms by which semantic cues shape reasoning remain poorly understood. We investigate this phenomenon in the context of comparative math problems with objective ground truth, revealing a consistent and directional framing bias: logically equivalent questions containing the words ``more'', ``less'', or ``equal'' systematically steer predictions in the direction of the framing term. To study this effect, we introduce MathComp, a controlled benchmark of 300 comparison scenarios, each evaluated under 14 prompt variants across three LLM families. We find that model errors frequently reflect linguistic steering, systematic shifts toward the comparative term present in the prompt. Chain-of-thought prompting reduces these biases, but its effectiveness varies: free-form reasoning is more robust, while structured formats may preserve or reintroduce directional drift. Finally, we show that including demographic identity terms (e.g., ``a woman'', ``a Black person'') in input scenarios amplifies directional drift, despite identical underlying quantities, highlighting the interplay between semantic framing and social referents. These findings expose critical blind spots in standard evaluation and motivate framing-aware benchmarks for diagnosing reasoning robustness and fairness in LLMs.

语言模型推理偏差公平性评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。