用思维链提炼记忆，让Mamba更好处理超长文本。
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation
- 用思维链摘要作为提示，教Mamba主动回忆长上下文。
- 在LONGMEMEVAL和HELMET上超越同类模型，长文本表现提升显著。
- 无需改架构，同时保持短文本处理能力，适合长序列任务。
Mamba理论上具备无限上下文能力，但在实际应用中，当序列长度远超训练长度时性能受限。本文提出一种简单有效的方法——回顾与推理（Recall with Reasoning, RwR），通过从教师模型蒸馏思维链（CoT）摘要，在微调阶段将这些摘要作为思维链提示前置，教会Mamba主动回忆并推理长上下文信息。在LONGMEMEVAL和HELMET数据集上的实验表明，RwR在相似预训练条件下，显著提升Mamba的长上下文表现，优于对比的Transformer及混合模型，且不改变原有架构，同时保持其短上下文处理能力。
原文摘要 · Abstract (English)
Mamba's theoretical infinite-context potential is limited in practice when sequences far exceed training lengths. This work explores unlocking Mamba's long-context memory ability by a simple-yet-effective method, Recall with Reasoning (RwR), by distilling chain-of-thought (CoT) summarization from a teacher model. Specifically, RwR prepends these summarization as CoT prompts during fine-tuning, teaching Mamba to actively recall and reason over long contexts. Experiments on LONGMEMEVAL and HELMET show RwR boosts Mamba's long-context performance against comparable Transformer/hybrid baselines under similar pretraining conditions, while preserving short-context capabilities, all without architectural changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。