arXiv:2606.31585cs.CVcs.AI2026-06被引 1

提出解耦相机位姿编码,解决多视角Transformer训练后期性能停滞问题。

DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers

论文配图:DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers
图 1 · 摘自论文原文
  • 将旋转与平移分开展现于位置编码中,避免参数混淆
  • 在大规模训练下实现稳定收敛,提升泛化能力
  • 适合需要高精度视图外推的3D生成任务

Transformer的卓越可扩展性使其广泛应用于3D计算机视觉,其中相机感知的位置编码对多视角几何提供空间线索至关重要。当前方法通常将相机参数(如外参或投影矩阵)作为相对位置编码嵌入注意力机制的查询、键和值向量中。然而,在扩展新视图合成(NVS)模型训练时,我们观察到训练后期性能显著停滞。本文探究其根本原因,发现将旋转和平移同时存于值向量的同一维度会引发二者独立识别的不确定性,制约训练可扩展性。为此,我们提出解耦姿态位置编码(DPPE),显式分离旋转与平移。在NVS任务上的大量实验表明,DPPE可在大规模训练设置下实现稳定长周期训练,并在视点外推(如增加视角数、缩放场景)中展现更优泛化性能。

原文摘要 · Abstract (English)

The remarkable scalability of Transformers has expanded their application to 3D computer vision, where camera-aware positional encoding is crucial for providing spatial cues in multi-view geometry. Recent advancements have established the practice of using camera parameters -- such as extrinsics or projection matrices -- as relative positional encoding into the query, key, and value vectors of the attention mechanism. However, when scaling up the training recipe of novel view synthesis (NVS) models with the camera-based positional encoding, we observe a significant issue: model performance stagnates in the late stages of training. In this paper, we investigate the cause of the performance bottleneck when scaling up and demonstrate that storing rotation and translation given by the positional encoding in the same dimensions of the value vector causes indeterminacy in their independent identification, hindering training scalability. To address this, we propose Decoupled Pose Positional Encoding (DPPE), a novel camera-based positional encoding that explicitly decouples rotation and translation. Extensive evaluations on NVS tasks demonstrate that DPPE enables stable long-term training even in scaled-up training setup. Furthermore, it exhibits superior generalization performance in extrapolation settings, such as handling an increased number of viewpoints and zoom-in scenarios.

3D生成多视角位置编码Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。