代码注释能显著提升跨语言翻译准确率,尤其描述整体功能的注释效果更佳。
Revisiting the Role of Natural Language Code Comments in Code Translation
- 对比带注释与无注释代码,大规模实验评估注释影响
- 关键注释使LLM翻译准确率最高提升近一倍
- 适合关注代码自动化转换与LLM应用的研究者
大型语言模型(LLMs)推动了编程语言间自动化代码翻译的新发展。由于多数代码专用LLM基于GitHub等仓库中注释丰富的代码进行预训练,推测自然语言注释可能有助于提升翻译质量。然而,现有代码翻译基准普遍缺少注释,导致其影响未被充分研究。本文开展大规模实证研究,分析注释对翻译性能的影响,涵盖超过80,000次翻译、1100多个代码样本,覆盖C、C++、Go、Java和Python五种语言间的两两翻译。结果表明,尤其是描述代码整体目的而非逐行功能的注释,能显著提升翻译准确性。基于此,我们提出COMMENTRA方法,证明其可使基于LLM的代码翻译性能提升近一倍。据我们所知,本研究是首个在规模、覆盖语言种类和综合性方面全面探索注释如何提升代码翻译准确率的工作。
原文摘要 · Abstract (English)
The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than $80,000$ translations, with and without comments, of $1100+$ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation. To the best of our knowledge, our study is the first in terms of its comprehensiveness, scale, and language coverage on how to improve code translation accuracy using code comments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。