用语义正确率评估代码翻译，发现传统指标无效
Beyond BLEU: A Semantic Evaluation Method for Code Translation
- 以程序执行结果判断翻译是否正确，而非字符串相似度
- LLM翻译语义正确率远高于传统启发式方法
- 适合关注代码生成质量的研究者和开发者
代码翻译是大模型的核心能力之一，但评估翻译正确性仍具挑战，因常用指标如BLEU仅衡量语法相似性，忽略程序语义。本文提出一种新型代码翻译评估方法，强调语义等价而非表面字符串相似。该方法将编译器测试技术应用于新领域，用于评估微调用于二进制提升任务（即反汇编为高级表示）的大模型。引入语义正确率，定义为产生正确执行结果的翻译比例，并通过评估基于LLM和启发式方法的反汇编器验证其有效性。结果表明，基于LLM的方法显著优于启发式方法，而BLEU得分与语义正确率相关性极低（r = -0.127 至 0.354），证明语法指标无法预测功能准确性。
原文摘要 · Abstract (English)
Code translation is one of the core capabilities of LLMs. However, evaluating the correctness of translations remains difficult, as commonly used metrics such as BLEU measure only syntactic similarity, disregarding program semantics. We propose a novel evaluation methodology for code translation tasks, emphasizing semantic equivalence over surface-level string similarity. Our approach applies established compiler testing methodology to a new domain, allowing the assessment of an LLM fine-tuned for binary lifting tasks (i.e. decompiling binaries to higher-level representations). We introduce a semantic correctness score, defined as the proportion of translations that produce correct execution outcomes, and demonstrate its application by evaluating LLM-based and heuristic decompilers. Our findings show that LLM-based approaches significantly outperform heuristic ones, while BLEU scores show negligible correlation with semantic correctness (r = -0.127 to 0.354), demonstrating that syntactic metrics fail to predict functional accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。