用智能选型替代堆叠，让视觉模型更省更准
Scope: Selective Cross-modal Orchestration of Visual Perception Experts
- 按每张图-文对动态选最优视觉编码器，不靠全量堆叠
- 用一个共享+一个选中的编码器，性能超四个全开的模型
- 推理计算量降低24%-49%，适合追求效率的视觉任务
视觉语言模型（VLMs）虽能从多个视觉编码器中受益，但简单堆叠导致收益递减且推理成本飙升。本文提出SCOPE，一种混合编码器（MoEnc）框架，通过实例级路由机制，为每个图像-文本对动态选择一个专用编码器，而非传统Token级路由。SCOPE包含一个共享编码器和一组可选编码器池，轻量级路由器利用文本提示与共享视觉特征间的交叉注意力，从可选池中选出最优编码器。为训练该路由器，引入双重熵正则化及辅助损失，平衡数据集级负载分布与实例级路由置信度。令人惊讶的是，仅使用一个共享加一个选中的编码器时，SCOPE的性能超过同时启用四个额外编码器的模型，同时计算量减少24%-49%。这表明，智能编码器选择优于暴力聚合，挑战了多编码器VLM的主流范式。
原文摘要 · Abstract (English)
Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49\%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。