提出VideoRoPE，让旋转位置编码更好处理视频时空结构。
VideoRoPE: What Makes for Good Video Rotary Position Embedding?
- 设计3D结构的VideoRoPE，兼顾时空对称性与独立索引。
- 在带干扰物的视觉寻针任务中显著优于已有方法。
- 适合长视频理解、生成等需要精准时序建模的任务。
尽管旋转位置编码（RoPE）及其变体因其长序列建模能力被广泛应用，但将其从一维扩展到具有复杂时空结构的视频仍具挑战。本文首次系统分析了视频中有效适配RoPE的四项关键特性，此前工作未充分考虑。为此，我们引入一个更具挑战性的V-NIAH-D任务，在原始视觉寻针任务中加入周期性干扰项，结果表明：缺乏合理时间维度分配的旧版RoPE变体极易被干扰误导。基于分析，我们提出 extbf{VideoRoPE}，其具有三维结构，包含低频时间分配以抑制周期振荡、对角布局保持空间对称性，以及可调时间间距以解耦时空索引。VideoRoPE在长视频检索、视频理解及视频幻觉等多样化下游任务中持续超越现有方法。代码将开源于https://github.com/Wiselnn570/VideoRoPE。
原文摘要 · Abstract (English)
While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce \textbf{VideoRoPE}, with a \textit{3D structure} designed to preserve spatio-temporal relationships. VideoRoPE features \textit{low-frequency temporal allocation} to mitigate periodic oscillations, a \textit{diagonal layout} to maintain spatial symmetry, and \textit{adjustable temporal spacing} to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at \href{https://github.com/Wiselnn570/VideoRoPE}{https://github.com/Wiselnn570/VideoRoPE}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。