发现大模型在排序偏好时严重不一致，易受无关选项干扰。
Measuring the Inconsistency of Large Language Models in Preferential Ranking
- 基于序理论定义一致性标准，如传递性、反身性等。
- 测试多款顶尖大模型，均无法满足一致性标准。
- 适合关注模型可信度与决策可靠性的研究者参考。
尽管大型语言模型（LLMs）取得显著进展，其偏见和幻觉问题依然存在，且在提供一致的序次偏好方面能力尚未被充分探索。本研究探讨了LLMs在密集决策空间或缺乏绝对答案场景下提供一致序次偏好能力。我们基于序理论提出一致性形式化定义，包括传递性、反对称性、可逆性及无关替代项独立性等准则。对若干前沿大模型的诊断实验表明，这些模型难以满足上述准则，表现出显著的位置偏差和较差的传递性，其偏好极易受无关替代项影响。研究揭示了大模型生成偏好排序中的严重不一致性，强调需进一步研究以解决这些局限。
原文摘要 · Abstract (English)
Despite large language models' (LLMs) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。