通过图像重建揭示Mamba长度泛化失效机制
Probing Length Generalization in Mamba via Image Reconstruction
- 用图像块序列重建任务控制实验，分析Mamba处理不同序列长度的行为
- 发现Mamba会根据训练长度分布调整策略，超出范围则性能下降
- 提出长度自适应变体，提升跨长度的重建表现，适合模型优化研究者
Mamba因其低计算复杂度和与Transformer相当的性能，成为广受关注的通用序列模型。然而，当推理序列长度超过训练时的范围时，其性能会下降。本文通过一个受控的视觉任务——从图像块序列重建图像——研究该现象。通过分析序列处理不同阶段的重构结果，我们发现Mamba会对其训练中遇到的序列长度分布产生定性适应，从而形成在该范围内有效但在超出范围时失效的策略。为支持这一分析，我们引入一种长度自适应的Mamba变体，在训练序列长度范围内显著提升了性能。研究结果为理解Mamba的长度泛化提供了直观视角，并指明了改进架构的方向。
原文摘要 · Abstract (English)
Mamba has attracted widespread interest as a general-purpose sequence model due to its low computational complexity and competitive performance relative to transformers. However, its performance can degrade when inference sequence lengths exceed those seen during training. We study this phenomenon using a controlled vision task in which Mamba reconstructs images from sequences of image patches. By analyzing reconstructions at different stages of sequence processing, we reveal that Mamba qualitatively adapts its behavior to the distribution of sequence lengths encountered during training, resulting in strategies that fail to generalize beyond this range. To support our analysis, we introduce a length-adaptive variant of Mamba that improves performance across training sequence lengths. Our results provide an intuitive perspective on length generalization in Mamba and suggest directions for improving the architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。