300M小模型通过优化训练策略，达到7B大模型的检索性能。
Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters
- 用硬负样本和多样化任务数据提升小模型检索能力
- 300M模型在检索任务上超越或媲美7B大模型
- 强调任务多样性比语言多样性对性能影响更大
训练有效的多语言嵌入模型面临语言多样性和任务目标差异的挑战。尽管小型多语言模型（<10亿参数）在多数多语言任务中表现良好，但在最主流的应用场景——检索任务中，始终落后于大型模型（>10亿参数）。这引出一个关键问题：能否通过针对性改造，让小模型在检索任务上实现性能跃升？本文研究了影响多语言嵌入效果的关键因素，包括训练数据规模、负样本策略和数据多样性。发现增加训练数据规模虽带来初期收益，但很快趋于饱和；引入硬负样本对持续提升检索准确率至关重要。进一步分析表明，训练数据中的任务多样性对性能提升的作用远超仅语言多样性。基于此，我们构建了一个约3亿参数的紧凑型多语言模型，在检索任务上达到甚至超越当前强大的70亿参数模型的水平。
原文摘要 · Abstract (English)
Training effective multilingual embedding models presents unique challenges due to the diversity of languages and task objectives. Although small multilingual models (<1 B parameters) perform well on multilingual tasks generally, they consistently lag behind larger models (>1 B) in the most prevalent use case: retrieval. This raises a critical question: Can smaller models be retrofitted specifically for retrieval tasks to enhance their performance? In this work, we investigate key factors that influence the effectiveness of multilingual embeddings, focusing on training data scale, negative sampling strategies, and data diversity. We find that while increasing the scale of training data yields initial performance gains, these improvements quickly plateau - indicating diminishing returns. Incorporating hard negatives proves essential for consistently improving retrieval accuracy. Furthermore, our analysis reveals that task diversity in the training data contributes more significantly to performance than language diversity alone. As a result, we develop a compact (approximately 300M) multilingual model that achieves retrieval performance comparable to or even surpassing current strong 7B models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。