arXiv:2505.20444cs.LGcs.CV2025-05NeurIPS被引 4

提出新型位置编码方法HoPE,提升视觉语言模型在长视频中的理解能力。

HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models

  • 设计混合频率分配策略,更好建模长序列时空关系。
  • 在4个视频基准上性能超越现有方法,最长支持1024帧输入。
  • 适合需要处理长视频的多模态任务,如视频检索与理解。

视觉语言模型(VLMs)在多模态任务中表现优异,但在长上下文场景下(尤其是长视频)性能下降明显。尽管旋转位置编码(RoPE)被广泛用于大语言模型的长度泛化,但将其扩展到视频的复杂时空依赖仍面临挑战。现有方法通常通过手动分配不同频率来编码3D位置信息,但缺乏理论支撑。本文首次系统分析不同分配策略对长上下文能力的影响,发现现有多模态RoPE难以可靠捕捉长距离语义相似性。为此,我们提出HoPE(Hybrid of Position Embedding),引入混合频率分配策略以实现任意长度上下文的稳定语义建模,并设计动态时间缩放机制,支持跨不同长度上下文的鲁棒学习与灵活推理。在四个视频基准上的实验表明,HoPE在长视频理解与检索任务中持续优于现有方法。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have made significant progress in multimodal tasks. However, their performance often deteriorates in long-context scenarios, particularly long videos. While Rotary Position Embedding (RoPE) has been widely adopted for length generalization in Large Language Models (LLMs), extending vanilla RoPE to capture the intricate spatial-temporal dependencies in videos remains an unsolved challenge. Existing methods typically allocate different frequencies within RoPE to encode 3D positional information. However, these allocation strategies mainly rely on heuristics, lacking in-depth theoretical analysis. In this paper, we first study how different allocation strategies impact the long-context capabilities of VLMs. Our analysis reveals that current multimodal RoPEs fail to reliably capture semantic similarities over extended contexts. To address this issue, we propose HoPE, a Hybrid of Position Embedding designed to improve the long-context capabilities of VLMs. HoPE introduces a hybrid frequency allocation strategy for reliable semantic modeling over arbitrarily long contexts, and a dynamic temporal scaling mechanism to facilitate robust learning and flexible inference across diverse context lengths. Extensive experiments across four video benchmarks on long video understanding and retrieval tasks demonstrate that HoPE consistently outperforms existing methods, confirming its effectiveness. Our code is available at https://github.com/hrlics/HoPE.

位置编码长视频多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。