用大模型生成多样化数据，训练更高效的小型检索器。
DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
- 用剪枝后的大模型生成多样训练数据，单阶段对比学习
- 小模型在多语言和长文本上表现优于传统检索器
- 适合追求高效且泛化强的检索系统开发者
大型语言模型（LLMs）在微调为密集检索器时表现出色，但其庞大的参数量带来显著推理延迟，包括大规模语料编码开销和查询延迟，限制了实际部署。相比之下，小型检索器虽效率更高，但在有限监督数据下泛化能力不足。本文提出DRAMA框架，利用LLMs训练更小、更具泛化能力的密集检索器。具体采用剪枝后的LLM作为主干，在单阶段对比学习中使用多样化LLM增强数据进行训练。实验表明，DRAMA在多语言和长上下文任务中优于传统编码器式检索器，并在多个任务和语言上实现优异性能，凸显将小型检索器训练与大模型进展结合的潜力，弥合效率与泛化之间的差距。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. However, their large parameter size brings significant inference time computational challenges, including high encoding costs for large-scale corpora and increased query latency, limiting their practical deployment. While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data. In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers. In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup. Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages. These highlight the potential of connecting the training of smaller retrievers with the growing advancements in LLMs, bridging the gap between efficiency and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。