评估大模型对印地语系语言的理解与生成能力，发现性能差异显著。
Analysis of Indic Language Capabilities in LLMs
- 基于训练数据、许可证和开发者信息分析28个支持印地语系语言的模型
- 印地语在模型中代表性最强，但其他语言性能差距大
- 研究结果可指导安全基准中印地语系语言的优先级选择
本报告评估了文本输入文本输出的大语言模型（LLMs）在理解与生成印地语系语言方面的能力，旨在识别并优先考虑适合纳入安全基准的印地语系语言。通过回顾现有评估研究与数据集，并分析28个支持印地语系语言的模型，研究考察了模型的训练数据、模型与数据的许可协议、访问方式及开发团队。对比不同评估数据集上的表现发现，各印地语系语言间存在显著性能差异。印地语在模型中覆盖最广，前五种语言的性能大致与使用者数量相关，但其后则表现不一。
原文摘要 · Abstract (English)
This report evaluates the performance of text-in text-out Large Language Models (LLMs) to understand and generate Indic languages. This evaluation is used to identify and prioritize Indic languages suited for inclusion in safety benchmarks. We conduct this study by reviewing existing evaluation studies and datasets; and a set of twenty-eight LLMs that support Indic languages. We analyze the LLMs on the basis of the training data, license for model and data, type of access and model developers. We also compare Indic language performance across evaluation datasets and find that significant performance disparities in performance across Indic languages. Hindi is the most widely represented language in models. While model performance roughly correlates with number of speakers for the top five languages, the assessment after that varies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。