arXiv:2606.20517cs.AIcs.PL2026-06被引 1

扩展代码生成评测基准至12种语言,揭示大模型跨语言能力短板。

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

论文配图:Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
图 1 · 摘自论文原文
  • 将原Python基准任务转化为12种语言等价任务,保持评估一致性。
  • 24个模型测试显示存在严重语言偏见和性能差异。
  • 支持未来更新自动追踪,适合研究跨语言代码生成的学者。

LiveCodeBench(LCB)已成为评估大语言模型(LLMs)代码生成能力的广泛采用基准,通过精选编程竞赛题、持续新增题目并按发布日期过滤,实现抗污染评估并全面衡量编码能力。然而LCB仍局限于Python,无法回答大模型在真实软件工程中所需多种语言间的泛化能力问题。我们提出Multi-LCB,一个涵盖十二种编程语言(含Python)的评测基准。Multi-LCB将原始LCB中的Python任务转换为其他语言等价任务,同时保留原有的污染控制与评估协议。由于完全兼容原LCB格式,Multi-LCB可自动追踪未来更新,系统性评估跨语言代码生成能力,并要求模型在超出Python范围外仍保持高性能。我们对24个指令与推理类大模型进行了评估,发现存在明显的Python过拟合、语言特异性污染及显著多语言性能差距。结果表明,Multi-LCB是严格的多语言代码评估新基准,直接解决LCB的核心局限,并暴露当前大模型能力的关键空白。

原文摘要 · Abstract (English)

LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.

代码生成多语言评测基准大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。