arXiv:2608.11891cs.CYcs.AI2026-08

评估印度大模型能力与评测成熟度,揭示真实差距与评测生态短板。

Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework

  • 基于公开数据对比8大能力域,构建评测成熟度指数(BMI)
  • 印度模型在传统基准表现强,但新领域参与度低且分布不均
  • 提醒政策制定者:能力差距可能源于评测生态不足

各国政府正加大对本土基础模型的投资以增强国家AI实力、数字主权和多语言计算能力。本文对印度基础模型生态系统进行基准驱动的对比评估,考察公开基准结果中的能力差距是否也反映了评测成熟度的不足。研究涵盖通用推理、编程与软件工程、智能体与计算机使用、网络安全、视觉与图像理解、视频与多模态理解、科研能力及印地语语言能力共八个领域,仅使用公开报告结果。提出一个探索性的四维基准成熟度指数(BMI),从标准化、参与度、独立验证和国家级覆盖四个维度评分。结果显示,印度模型在MMLU和MATH-500等成熟基准上得分较高,但这些基准已趋于饱和,前沿开发者不再报告。印度模型在新兴的智能体与领域专用评测中参与度极低,且组织间差异显著,Sarvam AI覆盖最广。BMI能修正纯描述性分析的判断,在部分领域重新评估成熟度。实践意义在于:当前许多看似的能力差距,可能源自评测生态的缺失,直接影响国家AI项目的监测与资助设计。原创性在于提出可复用的领域级评测生态成熟度评分工具,并首次应用于印度基础模型生态。

原文摘要 · Abstract (English)

Purpose: Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and multilingual computing. This paper assesses India's foundation-model ecosystem and examines whether apparent capability gaps in public benchmark evidence may also reflect gaps in evaluation maturity. Approach: The paper presents a structured, benchmark-based comparative assessment of Indian foundation models against global frontier and comparable-scale models across eight capability domains: general-purpose reasoning, coding and software engineering, agentic AI and computer use, cybersecurity, vision and image understanding, video and multimodal understanding, scientific research, and Indic language capability. Using only publicly reported results, it proposes an exploratory four-dimension Benchmark Maturity Index (BMI), scoring each domain on standardization, participation, independent verification, and national Findings: Indian models achieve strong scores on established benchmarks such as MMLU and MATH-500. However, these are now widely regarded as saturated, and frontier developers no longer report them. Indian models participate far less frequently in newer, agentic, and domain-specialized evaluations, and participation is highly uneven across organizations. Sarvam AI reports the broadest coverage by a substantial margin. The BMI refines, and in some cases revises, the maturity judgments a purely descriptive review would produce. Practical implications: Many apparent capability gaps cannot be distinguished, on available evidence, from evaluation-ecosystem gaps, with direct implications for how national AI programs should design monitoring and funding criteria. Originality: The paper proposes BMI as a reusable instrument for scoring evaluation-ecosystem maturity at the domain level and demonstrates its application to the Indian foundation-model ecosystem.

大模型评估评测成熟度印度AI基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。