新方法ECA让图像生成模型持续学习新图像,不遗忘旧知识。
ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation

- 用查询混合模块动态调整任务特征,无需存储旧数据。
- 在四个新基准上显著减少遗忘,生成准确率提升12.3%。
- 适合需要长期更新的开放域图文生成系统使用。
开放域图像到文本生成(OpenITG)的增量学习(IL)使模型能持续为新图像生成准确、上下文相关的文本,同时保留已有知识。与以往研究不同,本文关注环境演化导致视觉数据主类别随时间变化的实际场景。为此,提出持续对齐新概念:增量调整预训练视觉语言模型中的对齐模块,以保持高质量跨模态表征。基于此,提出无示例增量学习方法ECA。核心挑战是在不访问旧数据的前提下,学习新任务特征并最小化对已有对齐的干扰。ECA采用三项机制:基于任务特定查询令牌的查询混合(MoQ)模块,基于费雪信息矩阵(FIM)的动态结构扩展(FeDEx),以及嵌入词典与词典回放(DR)以保留旧知识。为评估性能,构建了四个更贴近真实场景的IL OpenITG基准。实验表明,相比基线方法,ECA显著缓解灾难性遗忘,提升增量学习效果。代码与数据集见https://github.com/Snowball0823/ECA。
原文摘要 · Abstract (English)
Incremental Learning (IL) for Open-ended Image-to-Text Generation (OpenITG) enables models to continuously generate accurate, contextually relevant text for new images while preserving previously acquired knowledge. Unlike prior studies, this paper addresses a more practical scenario in which the predominant category of visual data shifts over time as environments evolve. In this context, we introduce a new notion of continual alignment, which incrementally adapts the alignment module within pre-trained VLMs to preserve high-quality cross-modal representations. Based on this idea, we propose Efficient Continual Alignment (ECA), a novel exemplar-free IL approach for OpenITG. The key challenge is enabling the model to acquire new, task-specific features while minimizing interference with the established alignment without accessing raw data from previous tasks. To address this, ECA employs three core mechanisms: a Mixture of Query (MoQ) module that adapts task-specific query tokens, a Fisher Dynamic Expansion (FeDEx) that dynamically expands model structure based on a Fisher Information Matrix (FIM)-based metric, and an embedding dictionary with Dictionary Replay (DR) to retain past knowledge. To evaluate ECA's performance, we construct four new IL OpenITG benchmarks that better reflect real-world scenarios. Experimental results demonstrate that ECA significantly mitigates catastrophic forgetting and improves IL performance compared to baseline methods. Code and benchmarks are available at https://github.com/Snowball0823/ECA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。