用循环模型实现线性时间常数内存的文本嵌入，大幅降低长文本处理开销
Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models
- 采用垂直分块推理策略，使内存占用随输入长度增长趋于恒定
- 在多个基准上性能接近Transformer，但内存消耗显著更小
- 适用于Mamba2、RWKV、xLSTM等循环模型，通用性强
基于Transformer的嵌入模型存在二次计算复杂度和线性内存复杂度，限制了其在长序列上的应用。本文提出一种循环架构作为高效替代方案，引入垂直分块推理策略，使嵌入生成速度更快，且当输入长度超过垂直分块大小后，内存使用量趋于恒定。通过对Mamba2模型进行微调，验证其作为通用文本嵌入器的可行性，在多个基准测试中表现具有竞争力，同时相比Transformer模型内存占用显著减少。我们实证验证了该推理策略在Mamba2、RWKV和xLSTM模型上的适用性，确认不同架构间具有一致的运行时-内存权衡关系，表明循环模型是高效嵌入生成的有力替代方案。
原文摘要 · Abstract (English)
Transformer-based embedding models suffer from quadratic computational and linear memory complexity, limiting their utility for long sequences. We propose recurrent architectures as an efficient alternative, introducing a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size. By fine-tuning Mamba2 models, we demonstrate their viability as general-purpose text embedders, achieving competitive performance across a range of benchmarks while maintaining a substantially smaller memory footprint compared to transformer-based counterparts. We empirically validate the applicability of our inference strategy to Mamba2, RWKV, and xLSTM models, confirming consistent runtime-memory trade-offs across architectures and establishing recurrent models as a compelling alternative to transformers for efficient embedding generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。