FABLE是首个评估大模型数据流推理能力的基准，测试其在菜谱、行程等场景中的逻辑理解。
FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation
- 基于软件工程八类经典分析构建，覆盖数据流动态变化
- 三类模型中推理模型准确率最高，但速度慢20倍以上
- 适合研究程序理解、逻辑推理或评估模型可解释性的研究人员
理解数据如何移动、转换和持续存在（即数据流）是处理过程性任务的基础。尽管大语言模型（LLMs）在自然语言和编程语言上表现出色，但在涉及过程性任务决策时，尚未系统评估其数据流推理能力。本文提出FABLE，一个可扩展的基准，用于评估LLMs对结构化过程文本中的数据流理解。该基准改编了软件工程中的八类经典数据流分析：到达定义、非常忙表达式、可用表达式、活跃变量分析、区间分析、类型状态分析、污染分析和并发分析，并在烹饪食谱、旅行路线和自动化计划三个真实世界领域中实现。基准包含2400个问答对，每种领域-分析组合100个样本。我们评估了三类模型：专注推理的模型（DeepSeek-R1 8B）、通用模型（LLaMA 3.1 8B）和代码专用模型（Granite Code 8B）。每个提示采用五次采样完成的多数投票机制。结果显示，推理模型准确率更高，但推理速度比其他模型慢超过20倍；而通用模型与代码专用模型表现接近随机猜测。FABLE提供了首个诊断性基准，系统评估数据流推理能力，并为开发具备更强过程理解能力的模型提供洞见。
原文摘要 · Abstract (English)
Understanding how data moves, transforms, and persists, known as data flow, is fundamental to reasoning in procedural tasks. Despite their fluency in natural and programming languages, large language models (LLMs), although increasingly being applied to decisions with procedural tasks, have not been systematically evaluated for their ability to perform data-flow reasoning. We introduce FABLE, an extensible benchmark designed to assess LLMs' understanding of data flow using structured, procedural text. FABLE adapts eight classical data-flow analyses from software engineering: reaching definitions, very busy expressions, available expressions, live variable analysis, interval analysis, type-state analysis, taint analysis, and concurrency analysis. These analyses are instantiated across three real-world domains: cooking recipes, travel routes, and automated plans. The benchmark includes 2,400 question-answer pairs, with 100 examples for each domain-analysis combination. We evaluate three types of LLMs: a reasoning-focused model (DeepSeek-R1 8B), a general-purpose model (LLaMA 3.1 8B), and a code-specific model (Granite Code 8B). Each model is tested using majority voting over five sampled completions per prompt. Results show that the reasoning model achieves higher accuracy, but at the cost of over 20 times slower inference compared to the other models. In contrast, the general-purpose and code-specific models perform close to random chance. FABLE provides the first diagnostic benchmark to systematically evaluate data-flow reasoning and offers insights for developing models with stronger procedural understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。