为印度语言大模型设计多任务评估基准，推动本土化AI研究
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
- 基于MMLU Pro框架构建跨语言评测体系
- 覆盖9种印度主流语言，涵盖理解、推理与生成任务
- 适合关注南亚语言AI与文化适配的研究者
印度次大陆有超过15亿人使用印地语等印地语言，其丰富的文化遗产、语言多样性和复杂结构为自然语言处理带来独特挑战与机遇。IndicMMLU-Pro是一个全面的基准测试，用于评估大型语言模型（LLMs）在印地语言中的表现，基于MMLU Pro（大规模多任务语言理解）框架构建。该基准涵盖印地语、孟加拉语、古吉拉特语、马拉地语、卡纳达语、旁遮普语、泰米尔语、泰卢固语和乌尔都语等主要语言，针对印度语言的多样性特征设计，包含语言理解、推理和生成等多个任务，精心构建以捕捉印度语言的复杂性。IndicMMLU-Pro提供标准化评估框架，推动印地语言AI研究边界，促进更准确、高效且文化敏感的模型发展。本文阐述了基准的设计原则、任务分类体系及数据收集方法，并展示了前沿多语言模型的基线结果。
原文摘要 · Abstract (English)
Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。