用大模型评估英文请求类漏洞报告翻译效果，发现无模型全胜。
English Please: Evaluating Machine Translation with Large Language Models for Multilingual Bug Reports
- 对比DeepL、AWS Translate及ChatGPT等模型在漏洞报告上的翻译表现。
- ChatGPT语义翻译最佳，但源语言识别不如Claude和Mistral。
- 适合关注技术文本翻译与缺陷追踪系统集成的研究者。
准确翻译漏洞报告对全球软件开发协作至关重要。本研究首次全面评估机器翻译（MT）在漏洞报告上的表现，使用来自Visual Studio Code GitHub仓库的英文请求标签数据，比较DeepL、AWS Translate及ChatGPT、Claude、Gemini、LLaMA、Mistral等大模型。通过BLEU、BERTScore、COMET、METEOR、ROUGE等翻译评价指标，以及准确率、精确率、召回率、F1分数等分类指标，评估翻译质量与源语言识别准确性。结果表明，ChatGPT（gpt-4o）在语义与词汇层面翻译最优，但在源语言识别上未领先；Claude与Mistral的F1分数最高（分别为0.7182与0.7142），Gemini精确率最高（0.7414），AWS Translate在源语言识别准确率上表现最好（0.4717）。研究显示单一模型无法在所有任务中占优，强调需根据任务选择模型，并指出技术内容翻译需领域适配。代码与数据集已公开于GitHub。
原文摘要 · Abstract (English)
Accurate translation of bug reports is critical for efficient collaboration in global software development. In this study, we conduct the first comprehensive evaluation of machine translation (MT) performance on bug reports, analyzing the capabilities of DeepL, AWS Translate, and large language models such as ChatGPT, Claude, Gemini, LLaMA, and Mistral using data from the Visual Studio Code GitHub repository, specifically focusing on reports labeled with the english-please tag. To assess both translation quality and source language identification accuracy, we employ a range of MT evaluation metrics-including BLEU, BERTScore, COMET, METEOR, and ROUGE-alongside classification metrics such as accuracy, precision, recall, and F1-score. Our findings reveal that while ChatGPT (gpt-4o) excels in semantic and lexical translation quality, it does not lead in source language identification. Claude and Mistral achieve the highest F1-scores (0.7182 and 0.7142, respectively), and Gemini records the best precision (0.7414). AWS Translate shows the highest accuracy (0.4717) in identifying source languages. These results highlight that no single system dominates across all tasks, reinforcing the importance of task-specific evaluations. This study underscores the need for domain adaptation when translating technical content and provides actionable insights for integrating MT into bug-triaging workflows. The code and dataset for this paper are available at GitHub-https://github.com/av9ash/English-Please
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。