测试大模型真懂多少种语言，而非只会说表面话。
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
- 用精心设计的反例句检测语法能力，逼模型暴露真实语言理解水平。
- 翻译质量与训练数据量强相关，低资源语言表现明显落后但正逐步改善。
- 推理能力提升翻译，但对语法纠错效果不一，需按任务选配置。
多语言大模型部署于百余种语言，但现有评测多关注模型能否‘在某语言中完成任务’，混淆了流利度与语言掌握度。我们提出 M-GATE（多语言语法、翻译准确率与效率基准），覆盖30种类型多样、从高资源到低资源的语言。该基准包含三项任务：基于语言学家设计、对抗性选取的句子进行语法错误检测；29种目标语言对共享英文源文本的双向翻译，由三名LLM评委评分并经专业标注员验证；以及补充的分词器效率测量。我们在80多种配置下评估超过50个模型。结果表明，流利度与语言能力显著分离：能流畅翻译的模型在对抗性语法题上表现仅达随机水平，最高仅得0.36的马修斯相关系数（MCC），且错误倾向为漏报（接受错误句）而非误报。翻译质量与语言在预训练数据中的占比高度相关（对数通用爬虫数据占比的相关系数 r = 0.86），低资源语言存在陡峭惩罚，但随模型迭代正在缓解。启用推理能力可稳定提升翻译表现，但对语法检测影响较小，甚至对部分模型产生负向作用，因此最优配置依赖具体任务。为防数据污染，测试项私密存放于持续更新的公开排行榜后端，示例已发布（https://m-gate.ai）。
原文摘要 · Abstract (English)
Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency. We introduce M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency), a benchmark of linguistic proficiency spanning 30 typologically diverse languages from high- to low-resource. M-GATE comprises three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences that turn on hard, language-specific phenomena; round-trip translation of shared English sources across 29 target languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. We evaluate over 50 models in more than 80 configurations. Fluency and proficiency come apart sharply: models that translate competently sit near chance on the adversarial grammar items, the best reaching a Matthews correlation coefficient (MCC) of only 0.36, and their errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms. Translation quality closely tracks a language's share of pretraining data (r = 0.86 against log Common Crawl share), producing a steep low-resource penalty that is nonetheless narrowing with successive model releases. Enabling reasoning reliably improves translation, while its effect on error detection is smaller and for some models negative, so the best configuration is task-dependent. To resist contamination, test items are kept private behind a continuously updated public leaderboard, with illustrative examples released (https://m-gate.ai).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。