提出新型多模态位置编码,解决图像实例定位与跨模态对齐的模糊性问题。
RIG-RoPE: Relation-Stratified Multimodal Attention with Instance-Local Rotary Geometry and Representation-Aware Traversal Coordinates
- 设计实例局部旋转几何与关系分层注意力机制
- 实现对图像平移的完全不变性,保持空间关系一致性
- 适合需要稳定多模态对齐的视觉语言模型研究者
多模态旋转位置编码通过时间、高度和宽度相位对文本、图像和视频标记进行交错处理,但存在两个模糊性:跨实例的空间偏移依赖预处理选择,除非声明注册;视觉块的标量前进通常继承自坐标极值而非表示层面定义。本文提出RIG-RoPE,结合实例局部旋转几何、关系分层注意力与表示感知遍历坐标。RIG-RoPE分别归一化同关系得分,使用无高宽偏倚的LogSumExp统计量分配质量,并采用有序切片加法、并行空间尺度亚线性增长的遍历范围。文本按单位增量推进,图像块同时推进,视频则累积分词器时间标记。在匹配的仅推理Qwen2-VL-2B检查点实验中,原生与RIG纯文本路径完全一致。RIG对整图单图像平移及未注册图像对第二实例的平移均保持精确不变;原生注意力仍敏感于后者;高宽坍缩对照证实RIG保留同实例空间效应;视觉嵌入与所有参数未变。在三个种子的冻结小任务中,RIG零清洁到Gauge logits变化,而原始高宽基线在每个种子均有变化。Gauge准确率差异为+2/72、0、0,未通过预注册稳定性阈值。结果支持指定激活与Gauge机制,而非稳定性能提升、通用性或经验优越性。
原文摘要 · Abstract (English)
Multimodal rotary positional encodings apply temporal, height, and width phases to interleaved text, image, and video tokens. This creates two ambiguities: cross-instance spatial displacement depends on preprocessing chart choices unless registration is declared, and scalar advance across visual blocks is often inherited from coordinate extrema rather than defined at the representation level. We introduce RIG-RoPE, combining instance-local rotary geometry, relation-stratified attention, and representation-aware traversal coordinates. RIG-RoPE normalizes relation-homogeneous scores separately, allocates mass with a common H/W-neutral LogSumExp statistic, and uses traversal extent that is additive over ordered slices and sublinear over parallel spatial scale. Text advances by unit increments, image patches are simultaneous, and video accumulates over tokenizer temporal tokens. In a matched, inference-only Qwen2-VL-2B checkpoint experiment, native and RIG text-only paths were exactly equal. RIG was exactly invariant to a whole-chart single-image translation and to translating only the second instance of an unregistered image pair. Native attention remained sensitive to the latter, while an H/W-collapse control confirmed that RIG retained same-instance spatial effects; visual embeddings and all parameters were unchanged. Across three seeds of a frozen tiny task, RIG also had zero clean-to-Gauge logit change, whereas the raw-H/W baseline changed in every seed. Gauge-accuracy differences were +2/72, 0, and 0, failing the preregistered stability gate. These results support the specified activation and Gauge mechanisms, not stable task improvement, universality, or empirical superiority.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。