让视频中物体按新轨迹移动，保持3D运动关系不变
TrajectoryMover: Generative Movement of Object Trajectories in Videos
- 用合成数据生成管道构建成对视频数据，解决训练样本缺失问题
- 在多个视频上实现物体3D轨迹迁移，保留运动一致性与视觉真实感
- 适合想轻松编辑视频运动路径的非专业用户或动画创作者
生成式视频编辑已使非专业用户能轻松完成过去难以实现的短片编辑操作。现有方法主要关注在视频中指定物体的2D或3D运动轨迹，或改变物体/场景外观，同时保持视频的合理性和身份一致性。然而，尚缺乏一种在视频中移动物体3D运动轨迹的方法，即在保持相对3D运动关系的前提下移动物体。核心挑战在于此类场景缺乏成对视频数据。以往方法通常依赖巧妙的数据生成策略，从无配对视频构建合理成对数据，但当一对视频中的任一无法由另一生成时，该方法便失效。为此，我们提出TrajectoryAtlas——一个大规模合成成对视频数据的新生成管道，并基于此数据微调视频生成器TrajectoryMover。实验表明，该方法成功实现了物体轨迹的生成式移动。
原文摘要 · Abstract (English)
Generative video editing has enabled several intuitive editing operations for short video clips that would previously have been difficult to achieve, especially for non-expert editors. Existing methods focus on prescribing an object's 3D or 2D motion trajectory in a video, or on altering the appearance of an object or a scene, while preserving both the video's plausibility and identity. Yet a method to move an object's 3D motion trajectory in a video, i.e., moving an object while preserving its relative 3D motion, is currently still missing. The main challenge lies in obtaining paired video data for this scenario. Previous methods typically rely on clever data generation approaches to construct plausible paired data from unpaired videos, but this approach fails if one of the videos in a pair can not easily be constructed from the other. Instead, we introduce TrajectoryAtlas, a new data generation pipeline for large-scale synthetic paired video data and a video generator TrajectoryMover fine-tuned with this data. We show that this successfully enables generative movement of object trajectories. Project page: https://chhatrekiran.github.io/trajectorymover
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。