让大模型通过程序执行轨迹学习动态理解代码，提升生成准确率。
What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces
- 用真实程序执行轨迹训练模型，无需人工测试标注。
- 在CruxEval和MBPP上达到约80%的预测准确率。
- 动态中间计算比历史累积更适用于长序列执行。
代码生成与理解是大语言模型的关键能力。目前大多数LLM在代码数据上预训练和微调，但这些数据通常将代码视为静态字符串，很少利用其执行的动态信息。基于先前的轨迹建模工作，我们研究了执行微调（E.T.），一种显式建模真实程序执行轨迹的训练方法，无需人工测试标注。我们在不同粒度（行级和指令级）和策略下，对输出预测任务进行训练与评估，在CruxEval和MBPP上获得约80%的准确率，并发现动态沙盒（即模型自更新的独立中间计算）在长执行（最长达14,000步）中优于累积历史计算。最后，我们探讨了E.T.的实际应用价值。
原文摘要 · Abstract (English)
Code generation and understanding are critical capabilities for large language models (LLMs). Thus, most LLMs are pretrained and fine-tuned on code data. However, these datasets typically treat code as static strings and rarely exploit the dynamic information about their execution. Building upon previous work on trace modeling, we study Execution Tuning (E.T.), a training procedure in which we explicitly model real-world program execution traces without requiring manual test annotations. We train and evaluate models on different execution trace granularities (line and instruction-level) and strategies on the task of output prediction, obtaining around 80% accuracy on CruxEval and MBPP, and showing the advantages of dynamic scratchpads (i.e., self-contained intermediate computations updated by the model rather than accumulated as a history of past computations) on long executions (up to 14k steps). Finally, we discuss E.T.'s practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。