揭秘视频大模型如何在时间维度上推理，发现信息流动关键路径。
Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
- 通过可解释性分析，发现早期到中期层存在跨帧互动。
- 中间层逐步融合视频与语言信息，依赖时间概念对齐。
- 仅保留58%注意力边仍保持性能，适合优化模型效率。
视频大语言模型（VideoLLMs）将视觉-语言模型能力拓展至时空输入，支持视频问答等任务。尽管进展显著，其内部如何提取和传播视频与文本信息的机制仍不清晰。本文利用机制可解释性技术，分析了多种视频问答任务中的内部信息流。结果揭示出一致模式：（1）时间推理始于早期至中期层的活跃跨帧交互；（2）中期层实现视频与语言信息渐进融合，依赖视频表征与包含时间概念的语言嵌入对齐；（3）融合完成后，模型在中后期层生成正确答案。（4）基于此，我们发现模型通过选择有效信息路径、抑制大量注意力边（如LLaVA-NeXT-7B-Video-FT中减少58%）来维持视频问答性能。这些发现为理解视频大模型的时间推理机制提供了蓝图，并为提升可解释性与下游泛化能力提供实用洞见。项目主页及源码见https://map-the-flow.github.io
原文摘要 · Abstract (English)
Video Large Language Models (VideoLLMs) extend the capabilities of vision-language models to spatiotemporal inputs, enabling tasks such as video question answering (VideoQA). Despite recent advances in VideoLLMs, their internal mechanisms on where and how they extract and propagate video and textual information remain less explored. In this study, we investigate the internal information flow of VideoLLMs using mechanistic interpretability techniques. Our analysis reveals consistent patterns across diverse VideoQA tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame interactions in early-to-middle layers, (2) followed by progressive video-language integration in middle layers. This is facilitated by alignment between video representations and linguistic embeddings containing temporal concepts. (3) Upon completion of this integration, the model is ready to generate correct answers in middle-to-late layers. (4) Based on our analysis, we show that VideoLLMs retain their VideoQA performance by selecting these effective information pathways while suppressing a substantial amount of attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a blueprint for how VideoLLMs perform temporal reasoning and offer practical insights for improving model interpretability and downstream generalization. Our project page with the source code is available at https://map-the-flow.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。