arXiv:2503.18941cs.IRcs.CL2025-03中稿 · SIGIR 2025被引 9

探究生成式检索的训练与推理扩展规律,揭示模型规模、数据量和计算力的协同作用。

Exploring Training and Inference Scaling Laws in Generative Retrieval

  • 通过对比熵与生成损失设计新评估指标,实现跨方法性能连续比较。
  • 模型规模、训练数据量和推理计算力均显著提升生成式检索效果。
  • 解码器类大模型(如LLaMA)优于编码器类(如T5),适合构建高效检索系统。

生成式检索将检索任务重新定义为自回归生成过程,即大语言模型直接从查询生成目标文档。作为一项新范式,其性能与可扩展性的内在机制仍不明确。本文系统研究生成式检索的训练与推理扩展规律,探索模型规模、训练数据量与推理时计算资源如何共同影响性能。提出一种受对比熵和生成损失启发的新评估指标,提供连续性能信号,支持不同生成式检索方法间的稳健比较。实验表明,基于n-gram的方法高度符合训练与推理扩展规律。增大模型规模、扩大训练数据量及增加推理时计算资源均能提升性能,凸显三者互补性。在各类设置中,LLaMA模型始终优于T5模型,表明解码器架构在生成式检索中具有优势。研究结果强调模型规模、数据可用性与推理计算的协同作用,为未来系统的设计与优化提供新洞见。

原文摘要 · Abstract (English)

Generative retrieval reformulates retrieval as an autoregressive generation task, where large language models (LLMs) generate target documents directly from a query. As a novel paradigm, the mechanisms that underpin its performance and scalability remain largely unexplored. We systematically investigate training and inference scaling laws in generative retrieval, exploring how model size, training data scale, and inference-time compute jointly influence performance. We propose a novel evaluation metric inspired by contrastive entropy and generation loss, providing a continuous performance signal that enables robust comparisons across diverse generative retrieval methods. Our experiments show that n-gram-based methods align strongly with training and inference scaling laws. We find that increasing model size, training data scale, and inference-time compute all contribute to improved performance, highlighting the complementary roles of these factors in enhancing generative retrieval. Across these settings, LLaMA models consistently outperform T5 models, suggesting a particular advantage for larger decoder-only models in generative retrieval. Our findings underscore that model sizes, data availability, and inference computation interact to unlock the full potential of generative retrieval, offering new insights for designing and optimizing future systems.

生成式检索模型扩展大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。