用短剪辑视频训练模型,实现长视频中动作分割
Stitch Contrast and Segment_Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos
- 通过拼接剪辑视频构建多动作序列,模拟真实场景
- 设计对比学习任务,让模型理解动作时序上下文
- 仅需剪辑数据训练,却能处理未剪辑长视频
现有基于骨骼的动作分类模型依赖于精心剪裁的特定动作视频进行训练与测试,难以推广至真实场景。而当前的骨架动作分割模型虽可处理未剪裁视频,但需逐帧标注,成本高昂。本文提出一种新框架:仅使用短剪辑骨架视频训练,却能应用于更长的未剪裁视频。方法分三步:首先,通过时间拼接将剪辑视频作为基本动作单元,生成多动作组合序列;其次,设计新型对比预训练任务,使骨架编码器学习有意义的动作-时序上下文表征;最后,引入分割层完成动作分割。实验在剪辑源数据集与未剪辑目标数据集之间进行迁移评估,验证了方法在真实场景下骨架动作分割的有效性。
原文摘要 · Abstract (English)
Existing skeleton-based human action classification models rely on well-trimmed action-specific skeleton videos for both training and testing, precluding their scalability to real-world applications where untrimmed videos exhibiting concatenated actions are predominant. To overcome this limitation, recently introduced skeleton action segmentation models involve un-trimmed skeleton videos into end-to-end training. The model is optimized to provide frame-wise predictions for any length of testing videos, simultaneously realizing action localization and classification. Yet, achieving such an improvement im-poses frame-wise annotated skeleton videos, which remains time-consuming in practice. This paper features a novel framework for skeleton-based action segmentation trained on short trimmed skeleton videos, but that can run on longer un-trimmed videos. The approach is implemented in three steps: Stitch, Contrast, and Segment. First, Stitch proposes a tem-poral skeleton stitching scheme that treats trimmed skeleton videos as elementary human motions that compose a semantic space and can be sampled to generate multi-action stitched se-quences. Contrast learns contrastive representations from stitched sequences with a novel discrimination pretext task that enables a skeleton encoder to learn meaningful action-temporal contexts to improve action segmentation. Finally, Segment relates the proposed method to action segmentation by learning a segmentation layer while handling particular da-ta availability. Experiments involve a trimmed source dataset and an untrimmed target dataset in an adaptation formulation for real-world skeleton-based human action segmentation to evaluate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。