arXiv:2411.06151cs.CL2024-11EMNLP被引 4

用轻量模型实现伊斯兰文献多语言高效检索,开源免费。

Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust

  • 基于持续预训练与语言压缩,构建轻量化多语种检索模型。
  • 在资源受限设备上实现比通用大模型更快的搜索速度。
  • 适合宗教研究者、非营利机构及低预算团队使用。

大型语言模型(LLMs)显著提升了自然语言处理中的信息检索(IR)性能,但非商业驱动领域如宗教与文化遗产仍滞后于AI应用。伊斯兰文献具有重要文化价值,常被学者与公众使用,但缺乏统一的智能搜索资源。本文开发了一套面向伊斯兰领域的多语言非营利信息检索系统,面临多重挑战:特定语言数据稀缺、模型部署于资源受限设备、预算有限下的快速检索需求。通过持续预训练进行领域适配、语言缩减降低模型规模,构建出轻量级多语种检索模型,在性能上优于在通用数据上预训练的更大模型。评估表明,利用Rust语言特性设计的架构可在低资源环境下实现高效的语义搜索。

原文摘要 · Abstract (English)

The widespread use of large language models (LLMs) has dramatically improved many applications of Natural Language Processing (NLP), including Information Retrieval (IR). However, domains that are not driven by commercial interest often lag behind in benefiting from AI-powered solutions. One such area is religious and heritage corpora. Alongside similar domains, Islamic literature holds significant cultural value and is regularly utilized by scholars and the general public. Navigating this extensive amount of text is challenging, and there is currently no unified resource that allows for easy searching of this data using advanced AI tools. This work focuses on the development of a multilingual non-profit IR system for the Islamic domain. This process brings a few major challenges, such as preparing multilingual domain-specific corpora when data is limited in certain languages, deploying a model on resource-constrained devices, and enabling fast search on a limited budget. By employing methods like continued pre-training for domain adaptation and language reduction to decrease model size, a lightweight multilingual retrieval model was prepared, demonstrating superior performance compared to larger models pre-trained on general domain data. Furthermore, evaluating the proposed architecture that utilizes Rust Language capabilities shows the possibility of implementing efficient semantic search in a low-resource setting.

信息检索多语言轻量化开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。