提出新位置编码方式，让多视角注意力更适应3D场景几何。
RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
- 用射线投影坐标表示图像块位置，保证3D变换不变性。
- 自预测每块深度，实现对场景几何的自适应建模。
- 在新视角生成等任务中提升显著，尤其适合3D视觉任务。
我们研究多视角变换器中用于处理带姿态输入图像的补丁的位置编码，目标是实现唯一编码、支持SE(3)不变注意力及多频相似性，并能适应潜在3D场景几何。现有绝对或相对编码方案均无法满足这些要求，因此提出RayRoPE。该方法基于对应射线表示补丁位置，计算查询帧的投影坐标以确保SE(3)不变性；为适应场景几何，它无监督地预测每个补丁的深度以确定其沿射线的位置，同时建模不确定性并解析计算期望位置编码。我们在新视角合成、立体深度估计和前向3DGS重建任务上验证了方法，结果表明，尽管保持高效，RayRoPE仍持续优于其他位置编码方案（如在RE10K上LPIPS相对提升24%）。
原文摘要 · Abstract (English)
We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。