arXiv:2606.03879cs.CVcs.AI2026-06

重新训练5个视觉编码器组合,发现它们在多编码器模型中的作用不能简单靠掩码判断。

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

论文配图:Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
图 1 · 摘自论文原文
  • 通过重训练31种编码器组合,揭示其真实贡献度
  • 提出容量与必要性双维度评估,发现最优搭配非最强组合
  • 预投影器有效秩解释性能差异,指导高效配置设计

随着基础模型融合更多异构视觉流,理解多样化编码器在联合训练下的交互机制成为合理设计的前提。然而当前大规模视觉语言模型缺乏有效工具,且参数高效的编码器配置难以在训练前识别。为重新审视联合训练中编码器的角色,在16个基准的Cambrian-1套件上,我们采用统一流程对五种常见视觉编码器的所有31个非空子集进行重训练与评估(总计约20,000 GPU小时),得出三项发现:第一,从头重训所得编码器排名与固定检查点下掩码结果不同,包括整体排名第一的编码器也发生变化;第二,将每个编码器的贡献分解为两个维度——容量(单独使用时达到的分数)与必要性(移除时带来的下降),二者不可互换;最高容量的两个编码器组合表现不佳,而一个高容量锚点配合自适应互补编码器可媲美全五编码器模型,再增加编码器仅带来边际提升;第三,在固定参数量下,编码器预投影器的有效秩可解释残差得分变化。最强组合由能在联合训练中保持高秩的锚点与在该条件下秩扩张的互补编码器构成,表明更高秩、更少坍缩的投影器输入对应更优的编码器-投影器界面优化状态。综合容量-必要性分解与预投影器秩分析,结合全面重训练评估,揭示了多编码器视觉语言模型设计的方法论缺口,并提供了具体的改进范式。

原文摘要 · Abstract (English)

As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large vision-language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline (~20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder's contribution into two axes, Capacity, the score an encoder reaches on its own, and Necessity, the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank expands under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder-projector interface. Together, the Capacity-Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it.

多模态编码器评估模型设计视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。