不重放数据也能防止多模态大模型遗忘，通过保持注意力结构实现持续学习。
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
- 将跨模态注意力视为二维信号，用频谱统计量捕捉其尺度与方向特性。
- 在多个基准上表现优于现有方法，遗忘率降低20%以上，准确率提升显著。
- 适合需要长期适应新任务的多模态大模型应用，如智能客服、教育助手。
多模态大语言模型需持续适应不断变化的视觉领域、问题类型和用户指令，但持续微调常导致先前掌握的多模态能力严重遗忘。现有方法多依赖输出保留、数据重放或伪数据生成、嵌入几何正则化或任务专用参数分配，却难以控制内部跨模态注意力模式在适应过程中的漂移。本文提出无需重放的注意力谱正则化（ASR）框架，通过保留支持旧技能的注意力结构来缓解遗忘。ASR将交叉注意力图视为二维信号，提取其尺度和方向特性形成紧凑的频谱统计量，并仅存储技能级别的原型分布，而非重放图像-问题对、生成伪样本或旧阶段教师快照。后期通过相位不变的频谱正则项约束这些原型的有害漂移，同时允许实例级注意力适应新任务。理论分析表明，在频谱充分性假设下，技能级频谱漂移可控制遗忘，且傅里叶功率谱对空间平移和有界扰动具有稳定性。在持续性VQA和多模态指令调优基准（包括VQA v2、VQACL、CLT-VQA、CoIN、UCIT）上的实验显示，ASR在多项指标上均显著优于强基线方法，持续提升最终性能并大幅减少遗忘。保留技能级注意力结构是持续多模态大模型的有效轻量机制。代码已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions, yet continual fine-tuning often causes severe forgetting of previously acquired multimodal skills. Existing continual vision-language methods mainly preserve outputs, replay data or pseudo-data, regularize embedding geometry, or allocate task-specific parameters, but they provide limited control over how internal cross-modal attention patterns supporting old skills drift during adaptation. We propose Attention-Spectrum Regularization (ASR), a replay-free continual learning framework that preserves skill-conditioned structures of cross-modal attention. ASR treats cross-attention maps as two-dimensional signals, summarizes their scale and directional properties into compact spectral statistics, and stores only skill-wise prototype distributions instead of replaying past image-question pairs, generated pseudo-examples, or old-stage teacher snapshots. In later stages, a phase-invariant spectral regularizer constrains harmful drift of these prototypes while allowing instance-level attention to adapt to new tasks. We provide theoretical analysis showing that skill-conditioned spectral drift controls forgetting under a spectral sufficiency assumption, and that Fourier power spectra are stable to spatial translations and bounded perturbations. Experiments on continual VQA and multimodal instruction-tuning benchmarks, including VQA v2, VQACL, CLT-VQA, CoIN, and UCIT, show that ASR consistently improves final performance and reduces forgetting over strong replay-, regularization-, and adapter-based baselines. Preserving skill-level attention structure is an effective and lightweight mechanism for continual MLLMs. Code is available at https://github.com/Creative-zcx/attention-spectrum-replay
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。