动态调整模型适应能力,让视觉语言模型在复杂场景更聪明、简单场景不浪费。
MuRA: Multi-Rank Adaptation for Efficient and Effective Test-Time Vision-Language Generalization

- 根据图像复杂度动态分配不同容量的适配模块。
- 在多个跨数据集测试中达到顶尖准确率,计算开销更低。
- 适合需要高效部署的视觉语言模型应用者。
视觉语言模型虽具强大零样本能力,但在分布偏移下性能显著下降。现有测试时自适应(TTA)方法虽通过低秩适配实现参数高效,但存在根本瓶颈:依赖固定秩配置。由于视觉输入信息密度各异,固定秩导致复杂场景欠拟合、简单场景过拟合。为此,我们提出多秩适配(MuRA),基于标记级视觉复杂度动态选择并融合不同容量的适配模块。MuRA结合多秩正交分解实现优越且知识保留的初始化,并通过统一组件融合与连续路由更新持续学习语义到秩的映射。我们还提供严格的理论证明,验证该机制的必要性与梯度稳定性。关键的是,其动态设计在最深层视觉特征上表现最佳,利用最短梯度反传路径。大量实验表明,MuRA在广泛域泛化和跨数据集基准上达到当前最优准确率,同时显著降低计算与内存开销。
原文摘要 · Abstract (English)
Vision-language models exhibit remarkable zero-shot capabilities but suffer significant performance degradation under distribution shifts. While test-time adaptation (TTA) via Low-Rank Adaptation offers a parameter-efficient solution, we identify a fundamental bottleneck in current methods: the reliance on static rank configurations. Because visual inputs inherently possess varying information densities, a fixed rank forces an inevitable optimization compromise, leading to underfitting on complex scenes and overfitting on simple ones. To bridge this gap, we propose Multi-Rank Adaptation (MuRA), a novel framework that dynamically selects and fuses adaptation modules of varying capacities based on token-level visual complexity. MuRA synergizes Multi-Rank Orthogonal Decomposition to provide a superior, knowledge-preserving initialization, and Unified Component Fusion with Continuous Router Updating to sustainably learn semantic-to-rank mappings. Furthermore, we provide rigorous theoretical justifications mathematically proving the necessity and gradient stability of this adaptive mechanism. Crucially, MuRA's dynamic design uniquely thrives at the deepest visual layer, capitalizing on the shortest gradient backpropagation path. Extensive experiments demonstrate that MuRA achieves state-of-the-art accuracy across extensive domain generalization and cross-dataset benchmarks while significantly reducing both computational and memory overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。