通过可学习权重插值，让多模态模型持续学习时保持跨模态对齐。
AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
- 用插值方式融合两个冻结的模型检查点，统一调整参数。
- 在音视频图文检索任务上，各项指标均优于现有持续学习方法。
- 适合需要长期更新且不增加推理开销的多模态系统使用。
多模态模型如CLIP能构建跨模态共享嵌入空间以支持跨模态检索，但面对连续到达的新数据时，早期学到的跨模态对齐容易被破坏。传统持续学习方法返回单一检查点，使所有检索方向面临相同的稳定性-可塑性权衡。我们提出AlphaWiSE，一种后处理的权重空间插值方法，通过组合两个冻结的源检查点生成新模型。对于每个由检查点键识别的参数张量，AlphaWiSE为所有张量元素拟合一个共享的标量插值系数。该系数在较小的示例记忆库上训练，并用于生成最终的插值检查点。部署模型的结构和参数量与任一源检查点一致，无需额外推理开销。在音-图-文跨模态检索上的大量实验表明，该方法在多个检索方向和评估指标上均显著优于强基线持续学习方法。
原文摘要 · Abstract (English)
Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. The coefficients are fitted on a smaller exemplar memory and used to materialize one interpolated checkpoint. The deployed model has the same architecture and parameter count as either source checkpoint, which does not require additional inference time. Extensive experiments on audio-image-text retrieval show consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。