通过显式相似性度量提升动作分割精度,解决分类误差导致的边界不准问题。
Improving action segmentation via explicit similarity measurement
- 用多尺度特征与多个Transformer编码器生成帧级预测,通过相似性投票优化初始结果。
- 提出基于相邻帧特征相似性的边界修正算法,迭代调整边界位置,提升分割准确性。
- 支持有监督和无监督两种模式,适合视频动作分析、智能监控等场景使用。
现有监督动作分割方法依赖注意力机制或时序卷积进行帧级分类以捕捉时间依赖性。即使基于边界的检测方法也主要依赖初始帧级分类的准确性,低质量预测可能导致段落和边界的识别不精确。为此,本文提出ASESM(基于显式相似性度量的动作分割)方法,通过引入跨帧和预测间的显式相似性评估来提升分割精度。其监督学习架构以帧级多分辨率特征为输入,送入多个Transformer编码器,得到多个帧级预测后,通过相似性投票获得高质量初始预测。随后,采用新提出的边界修正算法,基于连续帧间特征相似性迭代调整边界位置。修正后的预测再经多阶段时序卷积进一步优化。作为后处理,可选地再次应用边界修正,并结合段落平滑方法,利用连续预测间的相似性剔除段内异常类别。此外,还提出一种完全无监督的边界检测-修正算法,仅依赖特征相似性识别边界,无需训练。在50Salads、GTEA和Breakfast数据集上的实验验证了有监督与无监督算法的有效性。代码与模型已开源于Github。
原文摘要 · Abstract (English)
Existing supervised action segmentation methods depend on the quality of frame-wise classification using attention mechanisms or temporal convolutions to capture temporal dependencies. Even boundary detection-based methods primarily depend on the accuracy of an initial frame-wise classification, which can overlook precise identification of segments and boundaries in case of low-quality prediction. To address this problem, this paper proposes ASESM (Action Segmentation via Explicit Similarity Measurement) to enhance the segmentation accuracy by incorporating explicit similarity evaluation across frames and predictions. Our supervised learning architecture uses frame-level multi-resolution features as input to multiple Transformer encoders. The resulting multiple frame-wise predictions are used for similarity voting to obtain high quality initial prediction. We apply a newly proposed boundary correction algorithm that operates based on feature similarity between consecutive frames to adjust the boundary locations iteratively through the learning process. The corrected prediction is then further refined through multiple stages of temporal convolutions. As post-processing, we optionally apply boundary correction again followed by a segment smoothing method that removes outlier classes within segments using similarity measurement between consecutive predictions. Additionally, we propose a fully unsupervised boundary detection-correction algorithm that identifies segment boundaries based solely on feature similarity without any training. Experiments on 50Salads, GTEA, and Breakfast datasets show the effectiveness of both the supervised and unsupervised algorithms. Code and models are made available on Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。