不需重训练，用时空分格提升视频大模型的视觉表征
Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding
- 通过分层时间分格捕捉多粒度时空交互
- 基于特征范数保留高信息量视觉区域，提升表征质量
- 即插即用，显著提升多个视频大模型性能
近期多模态大语言模型在视频理解任务上取得显著进展，但仍面临如何高效压缩视觉标记同时保持时空交互的挑战。现有方法如LLaVA系列采用简单池化或插值，忽视了视觉标记的复杂动态。为此，我们提出ST-GridPool，一种专为视频大模型设计的无训练视觉标记增强方法。该方法结合金字塔时间分格（PTG），通过分层时间分格捕获多粒度时空交互；以及基于范数的空间池化（NSP），利用标记范数与语义丰富度的相关性保留高信息区域。在多个基准上的大量实验表明，ST-GridPool无需重新训练即可持续提升视频大模型性能。该方法提供了一种高效且即插即用的视觉标记表示改进方案。代码已开源：https://github.com/bingjunluo/ST-GridPool。
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models (MLLMs) have significantly advanced video understanding tasks, yet challenges remain in efficiently compressing visual tokens while preserving spatiotemporal interactions. Existing methods, such as LLaVA family, utilize simplistic pooling or interpolation techniques that overlook the intricate dynamics of visual tokens. To bridge this gap, we propose ST-GridPool, a novel training-free visual token enhancement method designed specifically for Video LLMs. Our approach integrates Pyramid Temporal Gridding (PTG), which captures multi-grained spatiotemporal interactions through hierarchical temporal gridding, and Norm-based Spatial Pooling (NSP), which preserves high-information visual regions by leveraging the correlation between token norms and semantic richness. Extensive experiments on various benchmarks demonstrate that ST-GridPool consistently enhances performance of Video LLMs without requiring costly retraining. Our method offers an efficient and plug-and-play solution for improving visual token representations. Our code is available in https://github.com/bingjunluo/ST-GridPool.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。