arXiv:2603.01632cs.LGcs.AI2026-03AAAI被引 1

提出DeLo框架,解决多模态模型持续学习中的缺失模态问题。

DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning

  • 采用双分解低秩专家架构,分离模态间干扰
  • 在多个基准上超越现有方法,显著提升性能
  • 适合需要持续学习的多模态真实场景应用

将大模型应用于真实场景面临双重挑战:处理连续数据流的同时应对频繁的模态缺失,这一任务称为持续缺失模态学习(CMML)。现有方法多依赖提示调优,因可学习提示在共享嵌入空间中产生跨任务干扰而效果受限。直接使用共享模块的低秩适配(LoRA)也会因竞争梯度导致模态干扰。为此,我们提出DeLo,首个基于新型双分解低秩专家架构的CMML框架。该架构通过解耦的模态特定因子池动态组合秩一因子,生成低秩更新矩阵,有效缓解模态干扰。结合任务划分框架以防止灾难性遗忘,系统还包含跨模态引导路由策略处理不完整数据,以及任务无关的键值记忆实现高效推理。在多个标准CMML基准上的实验表明,本方法显著优于当前最优方案,凸显了结构化设计的低秩适配对真实世界多模态挑战的价值。

原文摘要 · Abstract (English)

Adapting Large Multimodal Models (LMMs) to real-world scenarios poses the dual challenges of learning from sequential data streams while handling frequent modality incompleteness, a task known as Continual Missing Modality Learning (CMML). However, existing works on CMML have predominantly relied on prompt tuning, a technique that struggles with this task due to cross-task interference between its learnable prompts in their shared embedding space. A naive application of Low-Rank Adaptation (LoRA) with modality-shared module will also suffer modality interference from competing gradients. To this end, we propose DeLo, the first framework to leverage a novel dual-decomposed low-rank expert architecture for CMML. Specifically, this architecture resolves modality interference through decomposed LoRA expert, dynamically composing LoRA update matrix with rank-one factors from disentangled modality-specific factor pools. Embedded within a task-partitioned framework that structurally prevents catastrophic forgetting, this expert system is supported by two key mechanisms: a Cross-Modal Guided Routing strategy to handle incomplete data and a Task-Key Memory for efficient, task-agnostic inference. Extensive experiments on established CMML benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches. This highlights the value of a principled, architecturally-aware LoRA design for real-world multimodal challenges.

多模态学习持续学习低秩适配缺失模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。