Mamba模型在文本重排序任务中表现媲美Transformer，适合长文本理解场景。
State Space Models are Strong Text Rerankers
- 用Mamba模型替代Transformer做文本重排序，利用其线性时间推理优势。
- Mamba-2在性能和效率上均优于Mamba-1，接近同规模Transformer表现。
- 适合需要长上下文建模且对推理速度敏感的检索系统应用。
Transformer在自然语言处理和信息检索中占据主导地位，但其推理效率低且难以外推到更长文本，促使研究者探索替代架构。状态空间模型（SSMs）如Mamba具有推理时复杂度为$O(1)$的优势。尽管潜力巨大，SSMs在需要细粒度查询-文档交互与长上下文理解的文本重排序任务中的表现仍不明确。本研究在多种规模、架构和预训练目标下，对比了基于SSM的Mamba-1和Mamba-2与Transformer模型在文本重排序任务上的性能与效率。结果表明：(1) Mamba架构在排名性能上可与同规模Transformer模型相当；(2) 其训练与推理效率低于使用flash attention的Transformer；(3) Mamba-2在性能与效率上均优于Mamba-1。这些结果证实了状态空间模型作为Transformer替代方案的潜力，并指出了未来信息检索应用中需改进的方向。
原文摘要 · Abstract (English)
Transformers dominate NLP and IR; but their inference inefficiencies and challenges in extrapolating to longer contexts have sparked interest in alternative model architectures. Among these, state space models (SSMs) like Mamba offer promising advantages, particularly $O(1)$ time complexity in inference. Despite their potential, SSMs' effectiveness at text reranking -- a task requiring fine-grained query-document interaction and long-context understanding -- remains underexplored. This study benchmarks SSM-based architectures (specifically, Mamba-1 and Mamba-2) against transformer-based models across various scales, architectures, and pre-training objectives, focusing on performance and efficiency in text reranking tasks. We find that (1) Mamba architectures achieve competitive text ranking performance, comparable to transformer-based models of similar size; (2) they are less efficient in training and inference compared to transformers with flash attention; and (3) Mamba-2 outperforms Mamba-1 in both performance and efficiency. These results underscore the potential of state space models as a transformer alternative and highlight areas for improvement in future IR applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。