发现视觉语言模型合集中的家族偏差并提出三种修复方法
Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles
- 按模型家族分层投票，提升错误相关样本的准确率
- 在三大数据集上，最高提升26个百分点，其中一项方法全胜基准
- 适合关注模型集成鲁棒性与实际部署效果的研究者
将来自不同提供商的视觉语言模型（VLMs）进行集成可最大化基准性能，但同架构家族的模型存在相关错误，标准投票机制无法处理。我们在 VQAv2、TextVQA 和 GQA 上研究了 17 个来自 8 个家族的 VLMs，发现家族相关错误使有效集成维度降至 2.5-3.6 个独立投票者，并在 1.5%-6.5% 的问题上形成误导层级，导致准确率降为 0%，即便最佳模型正确。为此提出三种家族感知方法：层次家族投票（HFV）在家族内聚合后跨家族投票，在误导层级上提升 18-26 个百分点；无训练的 QualRCCV 按校准度、家族质量与反家族规模加权，首次在所有三数据集上超越校准投票（p<0.05）；学习候选评分（LCS）训练交叉验证分类器，基于支持广度、家族多样性与模型质量重排序答案，实现最大增益：VQAv2 +0.68%、TextVQA +0.61%、GQA +2.45%，全部显著，且是唯一不会降低任一基准性能的学习方法。在 VQAv2 test-standard（EvalAI）上，使用 12 个模型时达到 87.83%，证实其泛化能力。
原文摘要 · Abstract (English)
Ensembling Vision-Language Models (VLMs) from different providers maximizes benchmark accuracy, yet models from the same architectural family share correlated errors that standard voting ignores. We study this structure across 17 VLMs from 8 families on VQAv2, TextVQA, and GQA. Family-correlated errors reduce effective ensemble dimensionality to 2.5-3.6 independent voters and create a Misleading tier (1.5-6.5% of questions) where correlated majority errors destroy accuracy to 0% despite the best model being correct. We propose three family-aware methods. Hierarchical Family Voting (HFV) aggregates within families before voting across them, recovering +18-26 pp on the Misleading tier. QualRCCV, a training-free method weighting models by calibration, family quality, and inverse family size, is the first to beat calibrated voting on all three benchmarks (p<0.05). Learned Candidate Scoring (LCS) trains a cross-validated classifier to re-rank candidate answers using support breadth, family diversity, and model quality, achieving the largest gains: +0.68% VQAv2, +0.61% TextVQA, +2.45% GQA -- all significant -- and is the only learned method that never degrades any benchmark. On VQAv2 test-standard (EvalAI), LCS reaches 87.83% with 12 models, confirming generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。