用沙普利值分析视频模型如何依赖文本，发现多选题本质是识破干扰项。
A Video Is Not Worth a Thousand Words
- 用沙普利值量化视频与文本特征贡献，可灵活定义模态层级。
- 6个模型在4个数据集上测试，均显示对文本高度依赖。
- 适合关注多模态模型公平性与解释性的研究者阅读。
随着视觉语言模型（VLMs）在理解世界中的应用日益广泛，相关研究致力于提升视频问答（VQA）数据集的难度和模型上下文长度。然而，大语言模型作为主干引发对文本主导的担忧，跨模态交互研究仍不充分。如何衡量多模态模型发展的方向是否正确？本文提出一种基于沙普利值的联合方法，可计算特征和模态得分，且特征与模态可任意定义。我们以6个不同上下文长度的VLM模型，在4个代表性数据集上进行多选题VQA对比，将视频帧与完整文本元素视为同等级特征，将多选题任务视为视频、问题与答案三模态的交互。结果表明模型严重依赖文本，并暴露多选题实质是识别干扰项的能力。代码已公开于https://github.com/sjpollard/a-video-is-not-worth-a-thousand-words。
原文摘要 · Abstract (English)
As we become increasingly dependent on vision language models (VLMs) to answer questions about the world around us, there is a significant amount of research devoted to increasing both the difficulty of video question answering (VQA) datasets, and the context lengths of the models that they evaluate. The reliance on large language models as backbones has lead to concerns about potential text dominance, and the exploration of interactions between modalities is underdeveloped. How do we measure whether we're heading in the right direction, with the complexity that multi-modal models introduce? We propose a joint method of computing both feature attributions and modality scores based on Shapley values, where both the features and modalities are arbitrarily definable. Using these metrics, we compare $6$ VLM models of varying context lengths on $4$ representative datasets, focusing on multiple-choice VQA. In particular, we consider video frames and whole textual elements as equal features in the hierarchy, and the multiple-choice VQA task as an interaction between three modalities: video, question and answer. Our results demonstrate a dependence on text and show that the multiple-choice VQA task devolves into a model's ability to ignore distractors. Code available at https://github.com/sjpollard/a-video-is-not-worth-a-thousand-words.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。