测试大模型对代码语义等价性的理解能力，发现准确率不足七成。
Assessing Code Understanding in LLMs
- 通过非平凡的代码变换测试模型理解能力
- 无上下文时准确率仅71%，有上下文时提升至79%
- 建议结合代码优化工具提升模型训练效果
我们对大型语言模型在代码理解方面的能力进行了实证评估，重点关注复杂的、保持语义不变的程序变换，如复制传播或常量折叠。研究发现，在未提供上下文的情况下，大模型在约41%的案例中无法正确判断语义等价性；当给出简单通用上下文时，错误率降至29%。为提高准确性，我们主张将大模型与代码优化工具结合，以增强训练并实现更鲁棒的程序理解能力。
原文摘要 · Abstract (English)
We present an empirical evaluation of Large Language Models in code understanding associated with non-trivial, semantic-preserving program transformations such as copy propagation or constant folding. Our findings show that LLMs fail to judge semantic equivalence in approximately 41\% of cases when no context is provided and in 29\% when given a simple generic context. To improve accuracy, we advocate integrating LLMs with code-optimization tools to enhance training and facilitate more robust program understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。