OSCAR通过在线软压缩与重排序，显著提升RAG推理速度且不损失准确率。
OSCAR: Online Soft Compression And Reranking
- 动态在推理时压缩检索内容，无需存储中间结果。
- 相比传统方法提速2-5倍，1B至24B模型均保持高精度。
- 适合需要高效知识增强的LLM应用，如实时问答系统。
检索增强生成（RAG）通过引入外部知识提升大语言模型的准确性与相关性，但随着检索规模增大，计算开销也随之增加。为解决此问题，我们提出OSCAR，一种查询相关的在线软压缩方法，在不牺牲性能的前提下降低计算负担。与传统硬压缩（缩短文本）或离线软压缩（将文档映射为连续嵌入）不同，OSCAR在推理时动态压缩检索信息，消除存储开销并支持更高压缩率。此外，我们还将OSCAR扩展为同时执行重排序，进一步优化RAG流水线效率。实验表明，该方法在1B至24B参数量的LLM上实现2-5倍推理加速，准确率几乎无损。模型已开源：https://huggingface.co/collections/naver/oscar-67d446a8e3a2551f57464295。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge, leading to improved accuracy and relevance. However, scaling RAG pipelines remains computationally expensive as retrieval sizes grow. To address this, we introduce OSCAR, a novel query-dependent online soft compression method that reduces computational overhead while preserving performance. Unlike traditional hard compression methods, which shorten retrieved texts, or soft compression approaches, which map documents to continuous embeddings offline, OSCAR dynamically compresses retrieved information at inference time, eliminating storage overhead and enabling higher compression rates. Additionally, we extend OSCAR to simultaneously perform reranking, further optimizing the efficiency of the RAG pipeline. Our experiments demonstrate state-of-the-art performance with a 2-5x speed-up in inference and minimal to no loss in accuracy for LLMs ranging from 1B to 24B parameters. The models are available at: https://huggingface.co/collections/naver/oscar-67d446a8e3a2551f57464295.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。