首个原生西班牙语高等数学推理基准，揭示模型跨语言能力差异
AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models
- 构建105道原生西班牙语高数题，覆盖7大领域并附逐步解题过程
- 顶尖模型在西班牙语下准确率超70%，但几何组合题仍难倒多数模型
- 证明原生语言评测能暴露翻译导致的推理盲区，适合多语言AI评估研究者
现有数学推理基准多为英语或翻译生成，易引入语义偏差并掩盖语言特异性错误。为此，我们提出AI4Math，一个包含105道原生西班牙语大学级数学题的基准数据集，覆盖代数、微积分、几何、概率、数论、组合与逻辑七大领域，每题均配有逐步人类解法。我们评估了六款大模型（GPT-4o、GPT-4o mini、o3 mini、LLaMA 3.3 70B、DeepSeek R1 685B、DeepSeek V3 685B）在零样本与思维链两种设置下的表现，分别在西班牙语和英语中进行。顶级模型（o3 mini、DeepSeek R1 685B、DeepSeek V3 685B）在西班牙语下准确率超过70%，而LLaMA 3.3 70B与GPT-4o mini低于40%。多数模型在双语间表现无显著差异，甚至GPT-4o在零样本西班牙语任务上表现更优。几何、组合与概率类题目对所有模型始终构成挑战。结果表明，原生语言基准对揭示标准指标未能捕捉的推理缺陷至关重要。
原文摘要 · Abstract (English)
Existing mathematical reasoning benchmarks are predominantly English only or translation-based, which can introduce semantic drift and mask languagespecific reasoning errors. To address this, we present AI4Math, a benchmark of 105 original university level math problems natively authored in Spanish. The dataset spans seven advanced domains (Algebra, Calculus, Geometry, Probability, Number Theory, Combinatorics, and Logic), and each problem is accompanied by a step by step human solution. We evaluate six large language models GPT 4o, GPT 4o mini, o3 mini, LLaMA 3.3 70B, DeepSeek R1 685B, and DeepSeek V3 685B under four configurations: zero shot and chain of thought, each in Spanish and English. The top models (o3 mini, DeepSeek R1 685B, DeepSeek V3 685B) achieve over 70% accuracy, whereas LLaMA 3.3 70B and GPT-4o mini remain below 40%. Most models show no significant performance drop between languages, with GPT 4o even performing better on Spanish problems in the zero shot setting. Geometry, Combinatorics, and Probability questions remain persistently challenging for all models. These results highlight the need for native-language benchmarks and domain-specific evaluations to reveal reasoning failures not captured by standard metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。