为波斯语设计的检索增强生成系统，提升低资源语言问答能力
PersianRAG: A Retrieval-Augmented Generation System for Persian Language
- 针对波斯语特点优化预处理、嵌入与检索流程
- 在多个波斯语基准数据集上验证了问答性能提升
- 适合需要本地化语言支持的生成系统开发者
检索增强生成（RAG）模型通过结合大规模预训练生成模型与外部检索机制，在多种自然语言处理任务中表现优异。然而，将RAG模型应用于波斯语这一低资源语言时面临独特挑战，主要涉及文本预处理、嵌入表示、检索策略、提示构造、语言建模及结果评估等多个环节。本文提出面向波斯语的真实世界RAG系统PersianRAG，针对上述问题设计创新解决方案，并在多个波斯语基准数据集上进行评估。实验结果表明，PersianRAG框架能有效提升波斯语问答任务的表现。
原文摘要 · Abstract (English)
Retrieval augmented generation (RAG) models, which integrate large-scale pre-trained generative models with external retrieval mechanisms, have shown significant success in various natural language processing (NLP) tasks. However, applying RAG models in Persian language as a low-resource language, poses distinct challenges. These challenges primarily involve the preprocessing, embedding, retrieval, prompt construction, language modeling, and response evaluation of the system. In this paper, we address the challenges towards implementing a real-world RAG system for Persian language called PersianRAG. We propose novel solutions to overcome these obstacles and evaluate our approach using several Persian benchmark datasets. Our experimental results demonstrate the capability of the PersianRAG framework to enhance question answering task in Persian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。