构建真实印度教育考题基准,评估大模型多语言推理能力
IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
- 用真实高阶考试题设计多语言评测框架
- 思维链提示使推理准确率显著提升,跨模型差异明显
- 中文读者可参考其多语言评估思路
大型语言模型的快速发展亟需反映真实学术严谨性与多语言复杂性的评估体系。本文提出IndicEval,一个可扩展的基准平台,用于评估大模型在英语和印地语双语环境下,针对印度公务员考试(UPSC)、理工科高考(JEE)和医学入学考试(NEET)的真实高阶试题表现,覆盖科学、技术、工程、数学及人文领域。与合成基准不同,IndicEval基于真实考试标准,实现对推理能力、专业领域知识及双语适应性的精准测量。框架采用零样本、少样本和思维链(Chain-of-Thought, CoT)提示策略自动评估,并支持新模型与新语言的模块化集成。在Gemini 2.0 Flash、GPT-4、Claude和LLaMA 3-70B上的实验揭示三大发现:第一,CoT提示显著提升推理准确性,跨学科与双语均有效;第二,各模型间性能差距显著,尤其在高复杂度考试中;第三,多语言退化仍是关键挑战,印地语表现相较英语明显下降,零样本条件下尤为突出。结果凸显当前大模型在双语推理与领域迁移方面仍存明显短板。整体而言,IndicEval为多语言教育场景下大模型的严谨、公平评估提供实践导向且可扩展的基础,并为提升推理鲁棒性与语言适配能力提供可操作洞察。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves reasoning accuracy, with substantial gains across subjects and languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third, multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in multilingual educational settings and offers actionable insights for improving reasoning robustness and language adaptability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。