用小模型复刻大模型的长尾查询重排能力，提升搜索效果。
RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine
- 设计高效标签生成流程，让小模型学大模型的语义理解能力。
- 提出编码器与解码器双路径训练方法，显著提升长尾查询重排效果。
- 在韩语搜索平台实测有效，适合需要精准长尾搜索的场景。
大型语言模型（LLMs）擅长理解查询与文档间的语义关系，尤其对长尾复杂查询表现优异。这类查询因用户反馈稀疏，传统基于反馈的排序方法效果差，因此LLM的排序能力极为宝贵。然而，大模型体积庞大、推理缓慢，亟需小型化高效模型（sLLMs）。现有蒸馏方法虽引入排序标签生成，但未能充分挖掘LLM潜力且流程繁琐。本文提出RRADistill：一种高效的标签生成管道与新颖的小模型训练方法，适用于编码器与解码器架构。我们引入编码器中的词项控制层以捕捉词项匹配信号，以及解码器中的排序层以增强理解能力。在韩国主流搜索平台进行A/B测试，验证了该方法在长尾查询重排上的有效性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. These queries are challenging for feedback-based rankings due to sparse user engagement and limited feedback, making LLMs' ranking ability highly valuable. However, the large size and slow inference of LLMs necessitate the development of smaller, more efficient models (sLLMs). Recently, integrating ranking label generation into distillation techniques has become crucial, but existing methods underutilize LLMs' capabilities and are cumbersome. Our research, RRADistill: Re-Ranking Ability Distillation, propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. We introduce an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding. A/B testing on a Korean-based search platform, validates the effectiveness of our approach in improving re-ranking for long-tail queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。