从视频中生成更长、更真实的密集轨迹,支持视频生成与编辑。
TrajLoom: Dense Future Trajectory Generation from Video
- 用网格锚点偏移编码减少位置偏差,提升轨迹表示精度。
- 预测长达81帧的未来轨迹,比现有方法延长三倍以上。
- 适用于视频生成、编辑任务,适合研究运动建模与可控生成者。
预测未来运动在视频理解与可控视频生成中至关重要。密集点轨迹是一种紧凑且表达力强的运动表示,但基于观测视频建模其未来演化仍具挑战。本文提出一个框架,从历史轨迹和视频上下文预测未来轨迹及可见性。方法包含三个组件:(1) 网格锚点偏移编码,通过将每个点表示为相对于像素中心锚点的偏移,降低位置依赖偏差;(2) TrajLoom-VAE,通过掩码重建与时空一致性正则化,学习密集轨迹的紧凑时空潜在空间;(3) TrajLoom-Flow,利用流匹配在潜在空间生成未来轨迹,结合边界提示与策略内K步微调实现稳定采样。此外,我们构建了TrajLoomBench,一个涵盖真实与合成视频的统一基准,采用与视频生成基准对齐的标准设置。相比当前最优方法,本方法将预测时长从24帧扩展至81帧,显著提升跨数据集的运动真实性与稳定性。生成的轨迹可直接用于下游视频生成与编辑。代码、模型检查点与数据集详见https://trajloom.github.io/。
原文摘要 · Abstract (English)
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。