arXiv:2607.15667cs.CV2026-07

用位置编码实现视频生成中的几何可控性,让视角变化更自然。

PE-Field 4D: Video Generation Models as Canvas

论文配图:PE-Field 4D: Video Generation Models as Canvas
图 1 · 摘自论文原文
  • 通过投影位置编码引导参考帧内容对齐目标视角
  • 在视角重定向任务中提升空间可控性,生成视频更连贯
  • 适合需要精准控制场景几何的视频编辑与新视角合成

扩散变压器在视频生成中表现优异,但控制视角变化下的场景几何仍具挑战。本文重新审视位置编码在视频扩散变压器中的作用,发现其能提供几何感知的空间偏差。若将参考标记按目标视角的投影位置编码,去噪模型会倾向于从输入视频的对应区域检索内容。基于此,我们提出一种几何感知交叉注意力机制,使目标视频潜在令牌能够关注来自参考图像或帧的结构化上下文令牌。为建立参考内容与目标相机轨迹的对应关系,上下文令牌采用结合目标视图2D重投影与深度感知消歧的投影位置编码方案。同时保留生成视频潜在变量原有的时空位置编码,实现几何引导注入而不破坏模型原有潜结构。该框架在视角依赖编辑任务中显著提升空间可控性,包括相机轨迹重播、新视角视频合成与几何感知视频编辑,同时保持底层视频扩散模型的生成先验。代码已公开于:https://github.com/MTLab/PE-Field。

原文摘要 · Abstract (English)

Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.

视频生成几何控制扩散模型视角编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。