用知识蒸馏把大模型变成高效交叉编码器，提升RAG重排序速度和效果。
Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
- 用小数据集微调80亿参数大模型，再4比特量化压缩
- 在特定问答任务上，准确率比传统方法高14%~21%
- 适合需要快速部署的实时检索增强生成场景
交叉编码器在检索增强生成（RAG）中能实现高精度重排序，但推理成本呈二次增长，难以实现实时部署。本文通过两阶段流程，将LLaMA 3（8B）微调为可直接替换的重排序器：首先使用Unsloth框架与LoRA适配器，在自建查询-文档相关性数据集上进行监督微调，随后采用4比特量化实现高效推理。该模型被集成到结合BM25与密集向量搜索的双检索器RAG管道中。在基于RAGAS框架的领域特定问答基准测试中，该微调后的LLaMA 3重排序器相较交叉编码器基线，在答案相关性上提升14%，上下文精确率提升16%，答案相似度提升19%，答案正确率提升21%，同时通过4比特量化显著降低推理开销。结果表明，指令微调的大模型可被转化为无需二次复杂度的高精度、高效率重排序器。
原文摘要 · Abstract (English)
Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using a two-stage pipeline: supervised fine-tuning on a custom query-document relevance dataset via the Unsloth framework with LoRA adapters, followed by 4-bit quantization for efficient inference. The resulting model replaces the cross-encoder in a dual-retriever RAG pipeline combining BM25 and dense vector search. Evaluated on a domain-specific question-answering benchmark using the RAGAS framework, our fine-tuned LLaMA 3 reranker achieves gains of 14% in answer relevancy, 16% in context precision, 19% in answer similarity, and 21% in answer correctness over the cross-encoder baseline, while reducing inference overhead through 4-bit quantization. These results demonstrate that instruction-tuned LLMs can be adapted into accurate, efficient rerankers without the quadratic complexity of traditional cross-encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。