让多模态模型更高效地学习跨模态关联，提升视觉语言任务表现。
CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
- 引入跨模态低秩适配路径，与模态内适配并行工作。
- 在多个基准上相比LoRA提升3%（视觉语言）和2%（音视频）。
- 首个支持多任务的轻量级多模态适配框架，适合资源受限场景。
基础模型推动了AI发展，但高效适配双流架构（如DINO、BERT）用于多模态下游任务仍具挑战。参数高效微调（PEFT）方法如低秩适配（LoRA）虽轻量，却仅在单模态内操作，难以捕捉跨模态交互。本文提出跨模态低秩适配（CoLA），通过在标准模态内适配路径外新增专用跨模态适配路径，实现双路径协同。该设计使未改写的单模态基础模型能有效适应多模态任务，且模态内与跨模态学习互不干扰。我们在多种视觉-语言（RefCOCO, RefCOCO+, RefCOCOg）和音频-视觉（AVE, AVS）基准上评估，结果表明CoLA持续优于LoRA，相对提升约3%和2%，同时保持参数效率。尤为关键的是，CoLA首次实现了视觉定位任务的多任务轻量化适配，填补了高效多模态适配的关键空白。代码已开源：https://github.com/peterwisu/CoLA
原文摘要 · Abstract (English)
Foundation models have revolutionized AI, but adapting them efficiently for multimodal tasks, particularly in dual-stream architectures composed of unimodal encoders, such as DINO and BERT, remains a significant challenge. ParameterEfficient Fine-Tuning (PEFT) methods like LowRank Adaptation (LoRA) enable lightweight adaptation, yet they operate in isolation within each modality, limiting their ability in capturing cross-modal interactions. In this paper, we take a step in bridging this gap with Cross-Modal LowRank Adaptation (CoLA), a novel PEFT framework that extends LoRA by introducing a dedicated inter-modal adaptation pathway alongside the standard intra-modal one. This dual-path design enables CoLA to adapt unimodal foundation models to multimodal tasks effectively, without interference between modality-specific and crossmodal learning. We evaluate CoLA across a range of vision-language (RefCOCO, RefCOCO+, RefCOCOg) and audio-visual (AVE, AVS) benchmarks, where it consistently outperforms LORA, achieving a relative gain of around 3% and 2%, respectively, while maintaining parameter efficiency. Notably, CoLA enables the first multitask PEFT framework for visual grounding, bridging a key gap in efficient multimodal adaptation. Code is available at https://github.com/peterwisu/CoLA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。