测试大模型在混用多语言提示时的代码生成能力
CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts
- 在英文提示中加入印地语、西班牙语、汉语拼音等混合语言
- 小模型在高混合度下准确率下降超30%
- 适合研究多语言编程场景下的模型鲁棒性
大型语言模型在代码生成任务中表现优异，广泛应用于代码补全、调试和编程辅助。然而，现有基准如HumanEval、MBPP和BigCodeBench主要评估纯英文提示下的表现，忽略了真实场景中多语言开发者常使用代码混合语言与模型交互的情况。为此，我们提出CodeMixBench，一个基于BigCodeBench的新基准，通过控制性代码混合（CMD）将三种语言对（印地语-英语、西班牙语-英语、汉语拼音-英语）融入提示的自然语言部分。我们全面评估了参数量从1.5B到15B的多种开源代码生成模型。结果表明，相比纯英文提示，代码混合提示持续降低Pass@1性能，且小模型在高CMD水平下下降更显著。CodeMixBench为多语言代码生成提供了真实评估框架，揭示了模型跨语言泛化的挑战与新方向。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable success in code generation tasks, powering various applications like code completion, debugging, and programming assistance. However, existing benchmarks such as HumanEval, MBPP, and BigCodeBench primarily evaluate LLMs on English-only prompts, overlooking the real-world scenario where multilingual developers often use code-mixed language while interacting with LLMs. To address this gap, we introduce CodeMixBench, a novel benchmark designed to evaluate the robustness of LLMs on code generation from code-mixed prompts. Built upon BigCodeBench, CodeMixBench introduces controlled code-mixing (CMD) into the natural language parts of prompts across three language pairs: Hinglish (Hindi-English), Spanish-English, and Chinese Pinyin-English. We comprehensively evaluate a diverse set of open-source code generation models ranging from 1.5B to 15B parameters. Our results show that code-mixed prompts consistently degrade Pass@1 performance compared to their English-only counterparts, with performance drops increasing under higher CMD levels for smaller models. CodeMixBench provides a realistic evaluation framework for studying multilingual code generation and highlights new challenges and directions for building robust code generation models that generalize well across diverse linguistic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。