通过双向推理任务评估模型对程序执行的因果理解。
The Path Not Taken: Duality in Reasoning about Program Execution
- 设计双任务评估框架：预测行为与反向推导输入修改
- 13个LLM在445个配对实例上表现差异显著，验证有效性
- 适合研究代码理解、模型推理能力的学者使用
大型语言模型在多种编程任务中表现出色，但其实际应用需真正理解程序执行过程，而非依赖表层模式。现有基准多聚焦特定输入下的程序属性预测（如代码覆盖率、程序输出），视角狭窄且易受数据污染影响。本文提出应通过两种互补推理任务考察程序执行的内在二元性：(i) 针对给定输入预测程序的实际行为；(ii) 推断输入需如何变异以达成特定行为目标。二者共同检验模型对执行流程的因果理解。我们构建了DexBench基准，包含445个配对实例，评估了13个LLM。结果表明，双路径推理能有效且区分性地衡量动态代码理解能力。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable capabilities across diverse coding tasks. However, their adoption requires a true understanding of program execution rather than relying on surface-level patterns. Existing benchmarks primarily focus on predicting program properties tied to specific inputs (e.g., code coverage, program outputs). As a result, they provide a narrow view of dynamic code reasoning and are prone to data contamination. We argue that understanding program execution requires evaluating its inherent duality through two complementary reasoning tasks: (i) predicting a program's observed behavior for a given input, and (ii) inferring how the input must be mutated toward a specific behavioral objective. Both tasks jointly probe a model's causal understanding of execution flow. We instantiate this duality in DexBench, a benchmark comprising 445 paired instances, and evaluate 13 LLMs. Our results demonstrate that dual-path reasoning provides a robust and discriminative proxy for dynamic code understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。