根据图像和问题特征,智能选择最适合的多模态模型。
LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?

- 用潜在状态匹配任务需求与模型能力,预测各模型表现。
- 在多个基准上优于固定模型和现有路由方法,尤其在复杂视觉任务中优势明显。
- 适合需要动态选型多模态模型的研究或工业应用。
多模态大语言模型在文字识别、图表理解、空间推理、视觉问答、成本和延迟方面能力各异。有效路由不仅需评估查询难度,还需将当前图像-问题输入的多模态需求与候选模型能力精准匹配。本文提出LatentRouter,将多模态模型路由建模为反事实多模态效用预测问题。给定图像-问题查询,该方法提取学习到的多模态路由胶囊,用模型能力令牌表示每个候选模型,并在这些状态间进行潜在通信,以估计若选择某模型其可能的表现。分布化输出头预测各模型的反事实质量,而有界胶囊校正机制在接近决策时优化结果,防止残差信号主导预测。基于效用的策略支持性能导向与性能-成本权衡路由,并通过共享模型评分与可用性掩码处理动态候选池变化。在MMR-Bench和VL-RouterBench上的实验表明,LatentRouter超越固定模型、特征级和学习型路由基线。额外分析显示,收益在依赖视觉、布局敏感或推理导向的任务组中最强,且潜在通信是主要提升来源。代码已开源:https://github.com/LabRAI/LatentRouter。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, LatentRouter extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。