构建首个覆盖9种印地语族语言的问答数据集，推动低资源语言AI发展。
IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages
- 基于SQuAD翻译构建，保持答案位置精准对齐
- 每种语言均含训练/验证/测试集，共覆盖9种语言
- 适合研究多语言NLP与低资源语言模型的学者
问答系统的发展主要惠及高资源语言，而印地语族语言尽管母语者众多，却长期缺乏支持。本文提出IndicSQuAD，一个涵盖九种主要印地语族语言的多语言抽取式问答数据集，源自SQuAD的系统性翻译。在前人基于马拉地语的MahaSQuAD工作基础上，本研究改进并扩展了翻译技术，确保跨语言语义保真度和答案跨度对齐。IndicSQuAD为每种语言提供完整的训练、验证与测试集，为模型开发奠定坚实基础。我们使用语言特定的单语BERT与多语言MuRIL-BERT进行基线评估，结果显示低资源环境下仍存在挑战。实验还揭示未来方向：拓展更多语言、构建领域专用数据集、引入多模态数据。数据集与模型已公开于https://github.com/l3cube-pune/indic-nlp。
原文摘要 · Abstract (English)
The rapid progress in question-answering (QA) systems has predominantly benefited high-resource languages, leaving Indic languages largely underrepresented despite their vast native speaker base. In this paper, we present IndicSQuAD, a comprehensive multi-lingual extractive QA dataset covering nine major Indic languages, systematically derived from the SQuAD dataset. Building on previous work with MahaSQuAD for Marathi, our approach adapts and extends translation techniques to maintain high linguistic fidelity and accurate answer-span alignment across diverse languages. IndicSQuAD comprises extensive training, validation, and test sets for each language, providing a robust foundation for model development. We evaluate baseline performances using language-specific monolingual BERT models and the multilingual MuRIL-BERT. The results indicate some challenges inherent in low-resource settings. Moreover, our experiments suggest potential directions for future work, including expanding to additional languages, developing domain-specific datasets, and incorporating multimodal data. The dataset and models are publicly shared at https://github.com/l3cube-pune/indic-nlp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。