语言影响代码评估结果，同一模型在不同语言中表现差异显著。
Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation
- 将评测提示本地化到五种语言，考察语言与模型的交互影响。
- GPT-4o在英语中表现最佳（44.72%），但阿拉伯语和印地语中Gemini更优（51.72%、53.22%）。
- 模型间一致性低，适合关注多语言评估公平性的研究者参考。
代码代理评估通常默认使用英语，但我们发现改变评判语言可逆转模型排名。将Agent-as-a-Judge提示体系本地化至五种语言（英语、阿拉伯语、土耳其语、中文、印地语），在三个开发代理框架下评估55个DevAI任务，共完成4950次评估。核心发现为模型与语言存在交互效应：GPT-4o在英语中满意度最高（44.72%），而Gemini在阿拉伯语（51.72%，p<0.001 vs GPT-4o）和印地语（53.22%）中领先。无单一模型在所有语言中占优，模型间对单个需求判断的一致性较低（Fleiss' κ≤0.231）。控制实验显示，仅本地化评判指令即可显著影响结果：印地语满意度从42.8%降至23.2%。表明语言应作为代理基准中的显式变量。完整需求级判断与运行时数据已公开以保障可复现性。
原文摘要 · Abstract (English)
Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings. We localize the Agent-as-a-Judge prompt stack to five typologically diverse languages (English, Arabic, Turkish, Chinese, Hindi) and evaluate 55 DevAI development tasks across three developer-agent frameworks and six judge backbones, totaling 4950 judge runs. The central finding is that backbone and language interact: GPT-4o achieves the highest satisfaction in English (44.72\%), while Gemini leads in Arabic (51.72\%, $p<0.001$ vs.\ GPT-4o) and Hindi (53.22\%). No single backbone dominates across all languages, and inter-backbone agreement on individual requirement judgments is modest (Fleiss' $κ\leq 0.231$). A controlled ablation further shows that localizing judge-side instructions, not just benchmark content, can be decisive: Hindi satisfaction drops from 42.8\% to 23.2\% under partial localization. These results indicate that language should be treated as an explicit evaluation variable in agentic benchmarks. Full requirement-level judgments and runtime statistics are released for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。