小规模检索增强生成，提升噪声搜索与领域迁移下的性能表现
Retro-li: Small-Scale Retrieval Augmented Generation Supporting Noisy Similarity Searches and Domain Shift Generalization
- 采用精准语义相似度搜索，在小规模非参数记忆库中找更优邻居
- 引入正则化机制，噪声搜索下困惑度降低，域偏移时泛化能力更强
- 可部署于模拟存内计算硬件，实现常数级搜索时间，性能损失<1%
检索增强生成（RAG）系统如Retro通过从包含万亿条目的非参数记忆库中检索，显著提升语言建模能力并降低毒性与幻觉。我们提出Retro-li，证明在小规模记忆库下检索依然有效，但需更精确的邻居搜索。为此，我们采用合适的语义相似度搜索，并首次引入对非参数记忆库的正则化：在推理时邻居搜索存在噪声的情况下显著降低困惑度，且在发生领域漂移时改善泛化能力。此外，Retro-li的非参数记忆库可潜在部署于模拟存内计算硬件，在实现O(1)搜索时间的同时引入检索邻居噪声，性能损失小于1%。代码已公开于https://github.com/IBM/Retrieval-Enhanced-Transformer-Little。
原文摘要 · Abstract (English)
The retrieval augmented generation (RAG) system such as Retro has been shown to improve language modeling capabilities and reduce toxicity and hallucinations by retrieving from a database of non-parametric memory containing trillions of entries. We introduce Retro-li that shows retrieval can also help using a small-scale database, but it demands more accurate and better neighbors when searching in a smaller hence sparser non-parametric memory. This can be met by using a proper semantic similarity search. We further propose adding a regularization to the non-parametric memory for the first time: it significantly reduces perplexity when the neighbor search operations are noisy during inference, and it improves generalization when a domain shift occurs. We also show that Retro-li's non-parametric memory can potentially be implemented on analog in-memory computing hardware, exhibiting O(1) search time while causing noise in retrieving neighbors, with minimal (<1%) performance loss. Our code is available at: https://github.com/IBM/Retrieval-Enhanced-Transformer-Little.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。