首个跨语言代码语义推理基准，通过执行验证确保结果真实可靠。
An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning
- 用程序执行轨迹验证替代静态分析或人工标注，确保语义恢复准确
- 覆盖3种语言10,583个真实项目，零样本测试最高F1达72.9%
- 适合研究代码理解、模型训练与可复现评估的开发者和研究人员
评估大语言模型能否恢复程序运行时的调用结构，而非仅生成通过测试的代码，仍是开放问题。现有基准多关注测试通过性，如HumanEval、MBPP、LiveCodeBench和SWE-Bench，虽有用但难以诊断模型对程序语义的理解程度。我们提出TraceEval，据知是首个执行验证的多语言代码语义推理基准：从源码恢复程序的运行时调用结构。不同于依赖静态工具或人工标注的调用图基准，TraceEval中每条正向边均由执行验证机械确认，消除标注争议与标签噪声。该基准包含（i）10,583个真实世界程序（2,129个测试，8,454个训练），来自1,600+开源仓库，涵盖Python、JavaScript和Java，通过LLM辅助的探测器生成流水线结合追踪验证获得；（ii）可复现的流水线，能将任意开源仓库转化为新的验证实例。我们在保留测试集上零样本评估10个LLM，最强模型Claude-Opus-4.6在三语言上平均F1为72.9%。为展示训练集作为监督基底的价值，我们基于其微调Qwen2.5-Coder系列：最高提升+55.6 F1，微调后Qwen2.5-Coder-32B达71.2%，仅比零样本Claude-Opus-4.6低1.7 F1。我们发布基准、流水线、基线及数据手册于https://github.com/yikun-li/TraceEva。
原文摘要 · Abstract (English)
Evaluating whether large language models (LLMs) can recover execution-relevant program structure, rather than only produce code that passes tests, remains an open problem. Existing code benchmarks emphasize test-passing outputs, from standalone programming tasks (HumanEval, MBPP, LiveCodeBench) to repository repair (SWE-Bench); this is useful, but offers limited diagnostic signal about which program semantics a model can recover from source. We introduce TraceEval, to our knowledge the first execution-verified, multi-language benchmark for code semantic reasoning: recovering a program's runtime call structure from source code. Unlike prior call-graph benchmarks that rely on static-tool output or hand-annotated ground truth, every positive edge in TraceEval is mechanically witnessed by validation execution, eliminating annotator disagreement and label noise for observed behavior. TraceEval consists of (i) 10,583 real-world programs (2,129 test, 8,454 train) extracted from 1,600+ open-source repositories across Python, JavaScript, and Java via an LLM-assisted harness-generation pipeline with tracer validation; and (ii) a reproducible pipeline that converts any open-source repository into new verified benchmark instances. We evaluate 10 LLMs at zero-shot on the held-out test split. The strongest model, Claude-Opus-4.6, reaches an average F1 of 72.9% across the three languages. To demonstrate the train split's utility as a supervision substrate, we fine-tune the Qwen2.5-Coder family on it: lifts of up to +55.6 F1 bring tuned Qwen2.5-Coder-32B to 71.2%, within 1.7 F1 of zero-shot Claude-Opus-4.6. We release the benchmark, pipeline, baselines, and a datasheet at https://github.com/yikun-li/TraceEva
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。