测试大模型对菜谱步骤顺序的推理能力，发现越长越乱越容易出错。
Evaluating LLMs' Reasoning Over Ordered Procedural Steps
- 用打乱的菜谱步骤重建正确顺序，评估模型逻辑排序能力。
- 序列越长、步骤越乱，模型准确率越低，最长达70步时性能显著下降。
- 提出三种新评估指标，更全面衡量顺序推理质量，适合研究者参考。
对程序性步骤的顺序推理是大语言模型的重要能力。本文研究从打乱的步骤中重构全局有序序列的任务，采用精心筛选的菜谱数据集，该领域中步骤顺序直接影响结果成败。在零样本和少样本设置下评估多个LLM，并提出一个综合评估框架，采用排名与序列对齐领域的成熟度量：肯德尔τ相关系数、归一化最长公共子序列（NLCS）和归一化编辑距离（NED），分别捕捉顺序质量的不同方面。分析显示，随着序列长度增加，模型性能下降，反映出更长流程带来的复杂性；输入中步骤位移越大（即打乱程度越高），性能进一步恶化。这些发现揭示了当前LLMs在处理长序列及高度无序输入时的局限性。
原文摘要 · Abstract (English)
Reasoning over procedural sequences, where the order of steps directly impacts outcomes, is a critical capability for large language models (LLMs). In this work, we study the task of reconstructing globally ordered sequences from shuffled procedural steps, using a curated dataset of food recipes, a domain where correct sequencing is essential for task success. We evaluate several LLMs under zero-shot and few-shot settings and present a comprehensive evaluation framework that adapts established metrics from ranking and sequence alignment. These include Kendall's Tau, Normalized Longest Common Subsequence (NLCS), and Normalized Edit Distance (NED), which capture complementary aspects of ordering quality. Our analysis shows that model performance declines with increasing sequence length, reflecting the added complexity of longer procedures. We also find that greater step displacement in the input, corresponding to more severe shuffling, leads to further degradation. These findings highlight the limitations of current LLMs in procedural reasoning, especially with longer and more disordered inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。