测试大模型执行复杂数据处理步骤的能力，发现其顺序敏感任务表现差。
CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes
- 构建了3462个真实数据处理任务，检验模型对步骤顺序和组合逻辑的遵守能力
- 10多个主流大模型在组合与顺序敏感任务中成功率骤降，最高失败率超80%
- 适合研究大模型推理可靠性、自动化数据处理系统的开发者参考
数据精炼涉及在不断演化的文本状态上执行多步操作，处理算子的组合方式和执行顺序共同决定最终结果。现有基准要么孤立文本编辑任务，要么将其与代码和工具执行混杂，尚不清楚大模型能否直接且忠实执行这类组合性、顺序敏感的数据精炼配方。为此，我们提出CDR-Bench，一个涵盖四个真实数据精炼领域、包含29种不同操作符的综合性基准，共3,462个高质量任务。该基准在原子、顺序无关和顺序敏感三种设置下评估模型，利用确定性参考输出实现精确评测。对10多个先进大模型的实验显示，性能在组合场景下显著下降，顺序敏感配方的成功率几乎归零。这些发现表明，当前大模型缺乏实现可靠组合数据精炼所需的程序忠实性。
原文摘要 · Abstract (English)
Data refinement involves executing multi-step recipes over evolving text states, where both composition and execution order of processing operators determine the outcome. While existing benchmarks either isolate text editing or entangle it with code and tool execution, it remains unclear whether LLMs can directly and faithfully execute these compositional, order-sensitive data refinement recipes. To fill this gap, we introduce CDR-Bench, a comprehensive benchmark featuring 3,462 high-quality tasks spanning four real-world data refinement domains and 29 distinct operators. Our benchmark evaluates models across atomic, order-agnostic, and order-sensitive settings, leveraging deterministic reference outputs to enable exact evaluation. Experiments on 10+ state-of-the-art LLMs reveal consistent failure patterns: performance degrades sharply in compositional settings, and order-sensitive recipe success collapses. These findings underline that current LLMs lack the procedural faithfulness required for reliable compositional data refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。