选对视角比堆更多模型更重要，能提升多视图融合效果
When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure
- 按核对齐增益排序冻结编码器，智能筛选最优视图组合
- 在五类识别任务中，平均提升AULC达5.8分，超越全融合方案
- 适合需要高效融合多源模型的科研与工程场景
基础模型库将多视图融合变为选择问题：从大量异构编码器池中，应融合哪些视图、融合多少？我们发现下游性能随融合编码器数量呈非单调变化；后期视图可能冗余或任务错配，导致准确率饱和甚至下降。我们形式化此问题为视图集组合，并提出KAGES（核对齐贪心编码器选择器），一种标签感知方法，通过中心化核-目标对齐的边际增益对冻结编码器排序。KAGES无需在选择过程中训练下游分类器，每个候选评估耗时为O(n²)，与编码器维度无关，且在单调性和正子模比条件下可提供(1−e⁻γ)的前缀保证。在五个识别范式及低样本、大池、全数据协议下，相比全融合，KAGES平均提升AULC 3.9、5.8和3.3点，优于DPP与设施选址选择。图像检索显示沿KAGES排序出现后期任务依赖性饱和，而峰值后下降现象也出现在冻结大语言模型融合中。结果表明，有效的大池融合依赖于选择紧凑且任务对齐的视图集合，而非盲目融合更多编码器。
原文摘要 · Abstract (English)
Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in $\mathcal{O}(n^2)$ time independent of encoder dimension, and admits a conditional $(1-e^{-γ})$ prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。