探索大模型在稀疏检索中的应用，发现编码-解码架构表现最佳。
Leveraging Decoder Architectures for Learned Sparse Retrieval
- 对比三种架构的稀疏表示能力，聚焦编码器、解码器和混合结构
- 编码-解码模型结合多标记解码，检索效果最优
- 大模型在零样本下表现受限，但解码器模型可随参数量提升超越
学习型稀疏检索（LSR）传统上集中于小规模仅编码器的Transformer架构。随着大规模预训练语言模型的发展，其在不同基于Transformer的架构（包括仅编码器、仅解码器和编码-解码模型）中生成稀疏表示进行检索任务的能力仍待深入探索。本研究系统考察了这些架构在LSR中的有效性，评估了多种稀疏表示头及模型规模的影响。结果表明，在零样本设置下，使用大型语言模型生成有效稀疏表示存在局限性，主要问题包括不恰当的词项扩展以及因缺乏扩展导致的性能下降。在三类骨干架构中，采用多标记解码策略的编码-解码架构表现最佳。尽管仅解码器模型整体表现弱于仅编码器模型，但在参数量极高时展现出超越潜力。
原文摘要 · Abstract (English)
Learned Sparse Retrieval (LSR) has traditionally focused on small-scale encoder-only transformer architectures. With the advent of large-scale pre-trained language models, their capability to generate sparse representations for retrieval tasks across different transformer-based architectures, including encoder-only, decoder-only, and encoder-decoder models, remains largely unexplored. This study investigates the effectiveness of LSR across these architectures, exploring various sparse representation heads and model scales. Our results highlight the limitations of using large language models to create effective sparse representations in zero-shot settings, identifying challenges such as inappropriate term expansions and reduced performance due to the lack of expansion. We find that the encoder-decoder architecture with multi-tokens decoding approach achieves the best performance among the three backbones. While the decoder-only model performs worse than the encoder-only model, it demonstrates the potential to outperform when scaled to a high number of parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。