对比相同动作视频中的细微差异，助力技能教学与评估。
Video Action Differencing
- 分三阶段构建智能流程：差异提议、关键帧定位、逐帧比对。
- 在549组视频对上识别出4469处精细差异，定位精度达2075个时间戳。
- 挑战大模型局部动作识别与细粒度帧对比能力，适合动作分析研究者。
两人执行相同动作时有何细微差别？本文提出视频动作差分（VidDiff）新任务，旨在识别同一动作视频间的微小差异，具有教练指导和技能学习等应用价值。为此，我们构建了VidDiffBench基准数据集，包含549对视频，含4469处人类标注的细粒度动作差异及2075个差异发生的时间戳。实验表明，GPT-4o、Qwen2-VL等主流多模态大模型在此任务上表现不佳。通过分析失败案例，我们指出两大挑战：跨视频相关子动作的定位与细粒度帧间比较。为此，我们提出VidDiff方法，采用代理式工作流，将任务分解为三个阶段：动作差异提议、关键帧定位、帧级差分，每个阶段使用专用基础模型。为推动该领域发展，我们已在Hugging Face发布数据集（https://huggingface.co/datasets/jmhb/VidDiffBench），代码开源（http://jmhb0.github.io/viddiff）。
原文摘要 · Abstract (English)
How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action, which has many applications, such as coaching and skill learning. To enable development on this new task, we first create VidDiffBench, a benchmark dataset containing 549 video pairs, with human annotations of 4,469 fine-grained action differences and 2,075 localization timestamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL. By analyzing failure cases of LMMs on VidDiffBench, we highlight two key challenges for this task: localizing relevant sub-actions over two videos and fine-grained frame comparison. To overcome these, we propose the VidDiff method, an agentic workflow that breaks the task into three stages: action difference proposal, keyframe localization, and frame differencing, each stage utilizing specialized foundation models. To encourage future research in this new task, we release the benchmark at https://huggingface.co/datasets/jmhb/VidDiffBench and code at http://jmhb0.github.io/viddiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。