用点追踪统一控制视频合成与运动，实现精准内容嵌入。
Go-with-the-Track: Video Compositing and Motion Control with Point Tracking
- 通过点轨迹联合参考图建模，实现跨帧精准定位
- 支持多参考图生成与相机运动控制，精度优于现有方法
- 适合影视制作、动画设计等需要精细控制的场景
电影制作需要精确的运动控制和参考图像合成——现有方法通常将二者分开处理。基于点追踪的图像到视频模型仅限于首帧内容插入，而参考到视频模型则缺乏对参考内容在各帧间融合过程的细粒度时空控制。我们提出 Go-with-the-Track，通过同时条件于多个参考图像和参考锚定的点轨迹，将两者能力统一。该方法将传统点轨迹扩展为显式建立生成帧与参考图像之间对应关系的机制，从而实现全程精准合成与运动控制。为此，我们引入空间感知的点轨迹嵌入，采用坐标分量独立的 MLP 结合时间池化，编码点轨迹全序列坐标。该表示既保留每个点轨迹的空间特征（作为唯一标识），又使嵌入相似性直接反映空间接近性，提升点轨迹区分与关联能力。通过轻量适配器将点轨迹嵌入注入视频扩散变换器，解决像素到补丁分辨率不匹配问题，避免了简单下采样导致的运动细节丢失。采用混合训练策略，在动态、静态及合成场景视频数据集上联合训练，增强运动可控性。实验表明，Go-with-the-Track 在单一模型中实现了优越的运动与参考控制，并支持新功能：多参考条件视频生成与点轨迹驱动的合成，以及静态与动态场景下的相机控制。
原文摘要 · Abstract (English)
Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks -- extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。