首个多语言编程评测基准，测试非英语开发真实难题
Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture
- 构建10种语言300个真实编程任务，覆盖国际化等非英语特有问题
- 最强模型仅63.1%通过率，多数任务无模型能解
- 揭示多语言编程能力是独立于通用编码能力的新维度
现有编程代理评估几乎全部基于英语，无法反映实际多语言部署场景。我们提出Terminal-Bench-LILT，一个包含300个真实编程任务的多语言评测集，涵盖阿拉伯语、捷克语、德语、西班牙语、印地语、日语、韩语、塞尔维亚语、土耳其语和中文十种语言。每项任务针对非英语软件开发中的特定问题，如国际化、编码格式、文本归一化及文化惯例，且无直接英文对应。所有任务由母语程序员编写，并通过多阶段质量控制验证。对六款前沿模型的评估显示，即使最强模型通过率也仅达63.1%，许多任务未被任何模型解决。性能随语言显著变化，且不与通用编码基准排名一致，表明多语言编程能力是独特且尚未充分探索的能力维度。样例任务见https://github.com/lilt/terminal-bench-lilt
原文摘要 · Abstract (English)
Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Terminal-Bench-LILT, a suite of 300 authentic coding tasks in ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. Each task targets issues specific to non-English software development that have no direct English equivalent, e.g., internationalization, encoding, text normalization, and cultural conventions. All tasks are authored by native-speaker programmers and validated through a multi-stage quality control pipeline. Evaluation of six frontier models reveals that even the strongest model reaches only 63.1\% pass rate, with many tasks unsolved by any model. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis. Sample tasks are available at https://github.com/lilt/terminal-bench-lilt
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。