针对第一视角视频编辑难题,构建了实时模型与评测基准。
EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing
- 设计专用数据集并开发实时流式编辑模型
- 在第一视角任务中显著提升编辑稳定性与指令遵循度
- 适合交互式AR应用与动作捕捉研究者使用
我们研究面向交互式AR应用的第一视角视频指令引导编辑。尽管现有AI视频编辑器在第三人称视频上表现良好,但第一视角视频因快速身体运动和频繁的手物交互带来显著领域差异。此外,现有离线编辑流程延迟高,难以支持实时交互。为此,我们构建了完整的第一视角视频编辑生态:首先,创建EgoEditData数据集,精心设计并人工标注,包含丰富手物交互且明确保留手部;其次,开发EgoEdit模型,支持单张GPU上的实时流式推理;最后,提出EgoEditBench评测套件,聚焦指令忠实性、手部与交互保留性、以及在身体运动下的时间稳定性。在第一视角和通用编辑任务中,EgoEdit均实现时间稳定、指令忠实的交互式结果。在第一视角基准上相较现有方法有明显提升,同时在通用任务上保持与最强基线相当的性能。EgoEditData与EgoEditBench将对社区公开。详见https://snap-research.github.io/EgoEdit
原文摘要 · Abstract (English)
We study instruction-guided editing of egocentric videos for interactive AR applications. While recent AI video editors perform well on third-person footage, egocentric views present unique challenges - including rapid egomotion and frequent hand-object interactions - that create a significant domain gap. Moreover, existing offline editing pipelines suffer from high latency, limiting real-time interaction. To address these issues, we present a complete ecosystem for egocentric video editing. First, we construct EgoEditData, a carefully designed and manually curated dataset specifically designed for egocentric editing scenarios, featuring rich hand-object interactions, while explicitly preserving hands. Second, we develop EgoEdit, an instruction-following egocentric video editor that supports real-time streaming inference on a single GPU. Finally, we introduce EgoEditBench, an evaluation suite targeting instruction faithfulness, hand and interaction preservation, and temporal stability under egomotion. Across both egocentric and general editing tasks, EgoEdit produces temporally stable, instruction-faithful results with interactive latency. It achieves clear gains on egocentric editing benchmarks-where existing methods struggle-while maintaining performance comparable to the strongest baselines on general editing tasks. EgoEditData and EgoEditBench will be made public for the research community. See our website at https://snap-research.github.io/EgoEdit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。