提出动态视觉令牌压缩方法，提升视频大模型效率与性能。
Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
- 设计动态压缩机制，按需保留关键视觉信息。
- 在VideoMME和MuirBench上分别提升2.7%和10.7%。
- 适用于长视频理解，适合追求高效推理的研究者。
大型视觉语言模型（LVLM）在图像与视频分析中应用广泛，但高质量视频数据集仍匮乏。现有视频大模型多基于单图模型扩展，难以高效处理长视频。本文构建了一个大规模合成数据集，通过精心设计的提示覆盖多样化问题，并提出一种动态视觉令牌压缩架构，在计算效率与性能间取得平衡。所提模型在多个视频任务中达到当前最优表现，尤其在多图理解任务中展现出优异泛化能力。在VideoMME上相比LLaVA-OneVision提升2.7%，在MuirBench上提升10.7%。代码已开源。
原文摘要 · Abstract (English)
The application of Large Vision-Language Models (LVLMs) for analyzing images and videos is an exciting and rapidly evolving field. In recent years, we've seen significant growth in high-quality image-text datasets for fine-tuning image understanding, but there is still a lack of comparable datasets for videos. Additionally, many VideoLLMs are extensions of single-image VLMs, which may not efficiently handle the complexities of longer videos. In this study, we introduce a large-scale synthetic dataset created from proprietary models, using carefully designed prompts to tackle a wide range of questions. We also explore a dynamic visual token compression architecture that strikes a balance between computational efficiency and performance. Our proposed \model{} achieves state-of-the-art results across various video tasks and shows impressive generalization, setting new baselines in multi-image understanding. Notably, \model{} delivers an absolute improvement of 2.7\% over LLaVA-OneVision on VideoMME and 10.7\% on MuirBench. Codes are available at https://github.com/Hon-Wong/ByteVideoLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。