让视频生成模型通过相机视角坐标精准控制运动轨迹。
SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers
- 用相机射线作为位置编码的第二维，替代传统网格坐标。
- 14B模型下旋转误差降29%，FVD降低43%，优于最强基线。
- 兼容预训练模型，新增参数不足0.1%，适合部署级应用。
视频扩散变换器通常基于像素-时间网格的位置索引进行标记，但这种索引并非世界坐标。我们真正需要的是标记所代表的世界点，它位于尚未生成的表面上，而其相机射线在用户指定轨迹后即固定。SCoPE因此将射线作为第二个位置坐标，使相机控制成为坐标系属性而非额外模块。该射线被加入预训练注意力机制的查询与键中，得分函数增加仅依赖两射线的项。其标准形式为线几何的倒数乘积，衡量两条视线接近交汇的程度。Normalize-Gate-Inject实现单一编码在度量与尺度不确定位姿源间的可训练性。该改造保持RoPE位精确，从不变的预训练DiT开始，新增参数少于0.1%。在Wan2.2数据集上，5B和14B模型在匹配数据与预算条件下，各项相机可控性与保真度指标均提升，闭环重访指标全面领先，且优势随模型规模扩大。14B模型下，旋转误差降低29%，FVD下降43%。
原文摘要 · Abstract (English)
Video diffusion transformers address their tokens by position on the pixel-time grid: an address in the tensor, not in the world. The address we would want, the world point a token depicts, lies on a surface not yet generated, while its camera ray is fixed once the user specifies a trajectory. SCoPE therefore treats the ray as a second positional coordinate, and camera control becomes a property of the coordinate system, not an added module. The ray is added to the pretrained attention's queries and keys, and the score gains a term that reads the two rays alone. Its canonical form, the reciprocal product of line geometry, measures how nearly two lines of sight meet. Normalize-Gate-Inject makes a single encoding trainable across metric and up-to-scale pose sources. The retrofit keeps RoPE bit-exact, starts from the unchanged pretrained DiT, and adds under 0.1/% new parameters. On Wan2.2 at 5B and 14B under matched data and budget, SCoPE improves every camera-controllability and fidelity metric, leads all closed-loop revisit metrics, and shows widening margins with model size. At 14B, rotation error falls 29/% and FVD 43/% below the strongest baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。