arXiv:2502.20395cs.LG2025-02ICML被引 12

测试时重路由提升多模态模型表现,无需重新训练。

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts

  • 测试时动态调整专家选择权重,优化推理过程。
  • 在多个挑战性任务上显著提升现有模型性能。
  • 无需训练参数,适合部署于已有模型的场景。

大型多模态模型(LMMs)中,视觉等非语言模态的感知能力通常弱于大语言模型(LLMs)的强大推理能力,限制了其在复杂下游任务中的表现。近期通过用混合专家(MoE)替换视觉编码器,提升了多模态表示的丰富性与多样性。然而,其性能高度依赖路由器对专家表示的加权与混合。我们发现,端到端训练的路由器并不能为每个测试样本生成最优路由权重。为此,提出一种新颖高效的测试时重路由方法 R2-T2:通过将测试样本的路由权重向邻近正确预测样本的权重方向局部优化,实现自适应调整。设计了三种基于不同优化目标与邻居搜索空间的 R2-T2 策略。R2-T2 在多种挑战性任务基准上持续且显著提升当前先进 LMM 性能,且不需更新任何基础模型参数。

原文摘要 · Abstract (English)

In large multimodal models (LMMs), the perception of non-language modalities (e.g., visual representations) is usually not on par with the large language models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on challenging downstream tasks. This weakness has been recently mitigated by replacing the vision encoder with a mixture-of-experts (MoE), which provides rich, multi-granularity, and diverse representations required by diverse downstream tasks. The performance of multimodal MoE largely depends on its router, which reweights and mixes the representations of different experts for each input. However, we find that the end-to-end trained router does not always produce the optimal routing weights for every test sample. To bridge the gap, we propose a novel and efficient method "Re-Routing in Test-Time (R2-T2)" that locally optimizes the vector of routing weights in test-time by moving it toward those vectors of the correctly predicted samples in a neighborhood of the test sample. We propose three R2-T2 strategies with different optimization objectives and neighbor-search spaces. R2-T2 consistently and greatly improves state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks, without training any base-model parameters.

多模态专家混合测试时优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。