测试大模型排序能力，发现其易出错且越复杂越差。
SortBench: Benchmarking LLMs based on their ability to sort lists
- 设计可扩展难度的SortBench基准测试排序任务
- 多数模型在长列表中丢漏或添加元素，准确率低于80%
- 测试时推理反而导致性能下降，非推理模型表现更稳
排序对人类和算法而言简单但对大语言模型（LLMs）却意外困难，因其难以忠实处理输入、进行逻辑比较及区分语法与语义。本文提出SortBench基准，涵盖多种难度级别，可灵活扩展。我们评估了七种先进LLM，包括当前测试时推理模型。结果表明，o3-mini虽整体表现较好，但在混合语法语义的输入（如按英文数字词排序）下仍会出错；所有模型在长列表上均出现丢项或增项问题，忠实度下降；测试时推理反而引发过度思考，导致性能退化；而无测试时推理的GPT-4o表现并不逊色。
原文摘要 · Abstract (English)
Sorting is a tedious but simple task for human intelligence and can be solved fairly easily algorithmically. However, for Large Language Models (LLMs) this task is surprisingly hard, as some properties of sorting are among known weaknesses of LLMs: being faithful to the input data, logical comparisons between values, and strictly differentiating between syntax (used for sorting) and semantics (typically learned by embeddings). Within this paper, we describe the new SortBench benchmark for LLMs that comes with different difficulties and that can be easily scaled in terms of difficulty. We apply this benchmark to seven state-of-the-art LLMs, including current test-time reasoning models. Our results show that while the o3-mini model is very capable at sorting in general, even this can be fooled if strings are defined to mix syntactical and semantical aspects, e.g., by asking to sort numbers written-out as word. Furthermore, all models have problems with the faithfulness to the input of long lists, i.e., they drop items and add new ones. Our results also show that test-time reasoning has a tendency to overthink problems which leads to performance degradation. Finally, models without test-time reasoning like GPT-4o are not much worse than reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。