构建日语运筹学评测集，检验大模型解决实际优化问题能力
JOR-Bench: Japanese Operations Research Benchmarks for Large Language Models
- 基于5个英文基准的翻译，创建1319道日语运筹学题目
- 强多语言模型在中日文间表现几乎无差别，平均准确率差仅-0.3个百分点
- 发现日语提示下模型易错答决策变量值而非目标值，提示设计需注意
我们提出JOR-Bench，一个用于评估大语言模型（LLMs）在运筹学（OR）问题建模与求解方面能力的日语基准测试集合。该集合包含五个现有英文基准的翻译：IndustryOR、MAMO Complex LP、NL4OPT、OptiBench 和 OptMATH，涵盖线性规划、混合整数规划、非线性规划和组合优化等领域的1319个问题。JOR-Bench为与求解器无关的基准，可兼容任意求解器或编程语言，由日语问题描述与期望数值答案成对组成。我们评估了七种LLM，包括通用多语言模型和日语专用模型，在原始英文版本与新日文版本上的表现，并进行跨语言对比。主评估中，我们使用OR-Tools的Python接口标准化执行流程，确保模型输出可比且可复现。结果显示，对于强大的多语言模型，建模能力基本不受语言影响，中英文平均准确率差异仅为-0.3个百分点。但错误分析揭示细微的跨语言差异，部分领域存在实用性的歧义消解失败，导致日语提示下模型输出决策变量值而非目标值。
原文摘要 · Abstract (English)
We present JOR-Bench, a collection of five Japanese-language benchmarks for evaluating the ability of large language models (LLMs) to formulate and solve operations research (OR) problems. Each benchmark is a Japanese translation of an existing English benchmark: IndustryOR, MAMO Complex LP, NL4OPT, OptiBench, and OptMATH, covering 1,319 problems spanning linear programming, mixed-integer programming, non-linear programming, and combinatorial optimization. JOR-Bench is a solver-independent benchmark that can be used with any solver or programming language, and consists of pairs of Japanese problem statements and expected numerical answers. We evaluate seven LLMs, including multilingual general-purpose models and Japanese-specialized models, on both the original English and the new Japanese versions, and compare performance across languages. For the main evaluation, we standardize execution with the Python interface to OR-Tools to make model outputs comparable and reproducible with open-source software. Our results show that OR formulation ability is largely language-neutral for strong multilingual models; the overall average accuracy difference between English and Japanese is only $-0.3$ pp. Yet error analysis reveals subtle cross-lingual differences, including a pragmatic disambiguation failure in some domains that causes models to output decision-variable values instead of the objective value when the prompt is in Japanese.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。