为视频大模型设计新型位置编码，提升时序理解与跨模态衔接能力。
VRoPE: Rotary Position Embedding for Video Large Language Models
- 提出VRoPE，通过均衡编码缓解注意力分布偏移。
- 在多个视频理解任务上显著优于传统RoPE方法。
- 适合需要精准时序建模的视频-语言模型研究者。
旋转位置编码（RoPE）在文本大模型中表现优异，但将其扩展至视频仍面临挑战，主要源于视频帧复杂的时空结构。现有方法如RoPE-3D虽尝试分别编码空间与时间维度，但仍存在注意力分布的位置偏差及视频-文本转换不连贯的问题。为此，本文提出针对视频大模型的新型位置编码方法——视频旋转位置编码（VRoPE）。该方法采用更均衡的编码策略，有效缓解注意力偏移，实现空间关注更均匀分布；同时重构位置索引结构，确保视频与文本标记间过渡平滑。在多种模型上的大量实验表明，VRoPE持续优于先前的RoPE变体，在视频理解、时序推理和检索任务中均取得显著提升。代码已开源：https://github.com/johncaged/VRoPE。
原文摘要 · Abstract (English)
Rotary Position Embedding (RoPE) has shown strong performance in text-based Large Language Models (LLMs), but extending it to video remains a challenge due to the intricate spatiotemporal structure of video frames. Existing adaptations, such as RoPE-3D, attempt to encode spatial and temporal dimensions separately but suffer from two major limitations: positional bias in attention distribution and disruptions in video-text transitions. To overcome these issues, we propose Video Rotary Position Embedding (VRoPE), a novel positional encoding method tailored for Video-LLMs. Specifically, we introduce a more balanced encoding strategy that mitigates attention biases, ensuring a more uniform distribution of spatial focus. Additionally, our approach restructures positional indices to ensure a smooth transition between video and text tokens. Extensive experiments on different models demonstrate that VRoPE consistently outperforms previous RoPE variants, achieving significant improvements in video understanding, temporal reasoning, and retrieval tasks. Code is available at https://github.com/johncaged/VRoPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。