研究不同语言提示对代码生成质量的影响，发现英语不总是最优。
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
- 用中、印、西、意四种语言翻译460个编程任务，构建多语言评测基准
- 英语提示在功能正确性和代码质量上并非始终领先，结果依赖模型与编程语言
- 生成代码常混用提示语种和英文，暴露语言偏好问题，适合多语言开发研究者
大型语言模型（LLMs）在不同自然语言提示下对同一编程任务的表现存在差异，这种现象称为语言偏差。尽管该现象在通用文本生成中被广泛研究，但在代码生成质量和编程规范方面仍缺乏深入探讨。本研究考察了GPT-4o mini、DeepSeek和Claude在中文、印地语、西班牙语和意大利语提示下的代码生成表现。我们设计了460个编码任务（各230个对应Python和Java），将原始英文提示精准翻译并人工校准为四种语言。通过测试通过率评估功能正确性，使用代码度量标准评估结构质量，结合静态分析工具检测问题，并分析标识符与注释中的语言使用情况。结果表明：（i）英语提示并不总能产生最高功能正确性或最佳代码质量；（ii）语言影响取决于编程语言和具体模型；（iii）生成代码频繁在注释和字符串字面量中混合提示语言与英语。本研究首次提供了一个经过精心构建的多语言代码生成基准，为提升多语言代码生成系统的鲁棒性提供了关键洞见。
原文摘要 · Abstract (English)
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。