探究Mamba模型在冻结状态下的语义表示能力，发现其输出存在严重结构缺陷。
Lost in State Space: Probing Frozen Mamba Representations
- 在固定分块边界提取输出，试图无须微调获得句子摘要
- 多任务测试显示其性能不如简单平均池化，且存在表示坍缩
- 提出正交注入改进机制，缓解状态空间的各向异性问题
Mamba的递归状态h_t本质上是迄今为止所有输入标记的压缩摘要。这引发一个诱人假设：若在固定分块边界提取标记级输出y_t，可免费获得语义句子摘要，无需池化头、无需微调、也无需[CLS] token。我们严格验证了该假设。在五个基准（SST-2、CoLA、MRPC、STS-B、IMDb）上，使用预训练Mamba-130M主干，在严格的冻结特征探测协议下，比较四种策略提取冻结句子表示，三组随机种子中计算可行时均进行测试。结果不支持该假设：分块边界读出未持续优于简单平均池化。我们识别并量化了两种结构性缺陷：严重各向异性（平均成对余弦相似度0.9999，标准差0.000044）和原始最终SSM状态的表示坍缩（在CoLA上所有三组种子下MCC=0.000，混淆矩阵确认）。我们进一步提出正交注入，一种修改后的递归机制，限制每步新信息注入。
原文摘要 · Abstract (English)
Mamba's recurrent state h_t is, by construction, a compressed summary of every token seen so far. This raises a tempting hypothesis: if we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free, with no pooling head, no fine-tuning, and no [CLS] token. We test this hypothesis carefully. Across five benchmarks (SST-2, CoLA, MRPC, STS-B, IMDb), we compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone under a strict frozen-feature probing protocol, using three random seeds where computationally feasible. The results do not support the hypothesis: patch boundary readouts do not consistently outperform simple mean pooling. We identify and quantify two structural pathologies: severe anisotropy (mean pairwise cosine similarity 0.9999, std 0.000044) and representational collapse in the raw final SSM state (MCC = 0.000 on CoLA across all three seeds, confirmed via confusion matrix). We further propose orthogonal injection, a modified recurrence that constrains new information per
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。