用统一模型同时完成动作分割与未来预测，效果优于专用模型。
ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
- 用可学习的令牌替代未来帧，让模型在训练中同时处理可见与不可见部分。
- 在50 Salads、Breakfast和GTEA数据集上均达到当前最优，分割与预测双双提升。
- 适合需要同时做动作识别与行为预判的场景，如智能监控或机器人交互。
时间动作分割与长期动作预测是视频动作时序分析中的两个重要任务。尽管二者具有明显关联性与互补潜力，但长期以来被作为独立任务研究。本文提出统一扩散模型ActFusion，联合解决这两个问题。核心思想是让模型在训练中一体化处理序列的可见部分（用于分割）与不可见部分（用于未来预测）。为此，我们引入新的预测性掩码策略：在训练时将视频后半段帧设为不可见，并用可学习的令牌替代，使模型学会推断未来内容。实验表明，动作分割与预测之间存在双向促进作用。ActFusion在50 Salads、Breakfast和GTEA三个标准数据集上均取得当前最佳性能，仅用单一统一模型就超越了针对每个任务专门设计的模型。
原文摘要 · Abstract (English)
Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation and action anticipation, jointly using a unified diffusion model dubbed ActFusion. The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner; the visible part is for temporal segmentation, and the invisible part is for future anticipation. To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future. Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation. ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。