arXiv:2602.10551cs.CVcs.AI2026-02中稿 · ICRA被引 3

改进3D多模态模型的位置编码,提升视觉特征的时空连续性与因果建模能力。

C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning

  • 设计三重混合位置索引,融合时间与二维空间坐标,保持视觉特征连续性。
  • 通过切比雪夫距离构建因果掩码,缓解长序列中早期视觉令牌被忽略的问题。
  • 在3D场景推理与视觉问答任务中显著优于基线模型,适合长序列多模态理解。

基于大语言模型的3D多模态模型(LMMs)已将3D视觉特征与语言表示对齐作为主流范式。然而,继承的旋转位置编码(RoPE)在多模态处理中存在局限:使用一维时间位置索引破坏了图像沿列方向的连续性,导致空间局部性丢失;且其假设时间上更接近的图像令牌具有更强因果关联,造成注意力分配随序列增长而长期衰减,使模型逐渐忽视早期视觉令牌。为此,我们提出C^2RoPE,一种显式建模视觉处理中局部空间连续性与空间因果关系的改进型RoPE。C^2RoPE引入时空连续位置嵌入机制,将一维时间位置与基于笛卡尔坐标的二维空间坐标结合,构造三元组混合位置索引,并采用频率分配策略对三个索引分量进行编码。此外,我们设计切比雪夫因果掩码,通过计算图像令牌在二维空间中的切比雪夫距离来确定因果依赖关系。在3D场景推理与3D视觉问答等多个基准测试上的评估结果表明C^2RoPE有效。代码已开源于https://github.com/ErikZ719/C2RoPE。

原文摘要 · Abstract (English)

Recent advances in 3D Large Multimodal Models (LMMs) built on Large Language Models (LLMs) have established the alignment of 3D visual features with LLM representations as the dominant paradigm. However, the inherited Rotary Position Embedding (RoPE) introduces limitations for multimodal processing. Specifically, applying 1D temporal positional indices disrupts the continuity of visual features along the column dimension, resulting in spatial locality loss. Moreover, RoPE follows the prior that temporally closer image tokens are more causally related, leading to long-term decay in attention allocation and causing the model to progressively neglect earlier visual tokens as the sequence length increases. To address these issues, we propose C^2RoPE, an improved RoPE that explicitly models local spatial Continuity and spatial Causal relationships for visual processing. C^2RoPE introduces a spatio-temporal continuous positional embedding mechanism for visual tokens. It first integrates 1D temporal positions with Cartesian-based spatial coordinates to construct a triplet hybrid positional index, and then employs a frequency allocation strategy to encode spatio-temporal positional information across the three index components. Additionally, we introduce Chebyshev Causal Masking, which determines causal dependencies by computing the Chebyshev distance of image tokens in 2D space. Evaluation results across various benchmarks, including 3D scene reasoning and 3D visual question answering, demonstrate C^2RoPE's effectiveness. The code is be available at https://github.com/ErikZ719/C2RoPE.

3D生成位置编码多模态模型因果建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。