显式引入物体信息能显著提升视频语言理解性能。
How Can Objects Help Video-Language Understanding?
- 用任意视觉算法提取物体结构化表示并融入多模态大模型
- 量化物体信息为文本后表现最优，且数据效率高
- 适合希望提升视频理解能力的研究者和开发者
多模态大语言模型是否仍需显式表示物体？一方面，预训练编码器将图像转为视觉标记，隐式建模物体与时空关系；另一方面，仅靠图像描述也能在理解任务中取得优异表现，尽管缺乏细粒度时空信息。为回答该问题，我们提出ObjectMLLM框架，可灵活利用任意计算机视觉算法提取并整合结构化视觉表示。在六个视频问答基准上的广泛评估表明，显式集成以物体为中心的表示依然必要。令人意外的是，将连续的结构化物体信息量化为纯文本表示的方法表现最佳，提供了一种高效的数据融合方式，便于集成其他视觉感知模块。代码与模型已开源。
原文摘要 · Abstract (English)
Do we still need to represent objects explicitly in multimodal large language models (MLLMs)? To one extreme, pre-trained encoders convert images into visual tokens, with which objects and spatiotemporal relationships may be implicitly modeled. To the other extreme, image captions by themselves provide strong empirical performances for understanding tasks, despite missing fine-grained spatiotemporal information. To answer this question, we introduce ObjectMLLM, a framework capable of leveraging arbitrary computer vision algorithm to extract and integrate structured visual representation. Through extensive evaluations on six video question answering benchmarks, we confirm that explicit integration of object-centric representation remains necessary. Surprisingly, we observe that the simple approach of quantizing the continuous, structured object information and representing them as plain text performs the best, offering a data-efficient approach to integrate other visual perception modules into MLLM design. Our code and models are released at https://github.com/brown-palm/ObjectMLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。