融合高低帧率数据,用正弦编码提升动作分割精度
Multi-Modal Graph Convolutional Network with Sinusoidal Encoding for Robust Human Action Segmentation
- 用正弦编码处理3D骨骼坐标,增强空间表示鲁棒性
- 多模态时序图融合模块对齐不同帧率数据,减少碎片化
- 引入平滑标签混合增强训练,降低过度分割错误
准确的时序动作分割对协作场景中的智能机器人至关重要,需精确理解子活动标签及其时间结构。然而,人体姿态估计和目标检测中的固有噪声常导致过度分割,破坏动作序列连贯性。为此,本文提出多模态图卷积网络(MMGCN),融合低帧率(如1 fps)视觉数据与高帧率(如30 fps)运动数据(骨骼与物体检测),缓解分割碎片化问题。框架包含三项关键贡献:首先,采用正弦编码将3D骨骼坐标映射至连续正弦-余弦空间,提升空间表征鲁棒性;其次,设计时序图融合模块,通过分层特征聚合对齐多模态输入的不同分辨率;第三,受人类动作自然平滑过渡启发,提出SmoothLabelMix数据增强方法,混合输入序列与标签生成具有渐变动作过渡的合成训练样本,增强预测的时间一致性,减少过度分割伪影。在双人动作数据集(Bimanual Actions Dataset)上的大量实验表明,该方法优于现有最先进模型,尤其在动作分割精度上表现突出,达成F1@10: 94.5% 和 F1@25: 92.8%。
原文摘要 · Abstract (English)
Accurate temporal segmentation of human actions is critical for intelligent robots in collaborative settings, where a precise understanding of sub-activity labels and their temporal structure is essential. However, the inherent noise in both human pose estimation and object detection often leads to over-segmentation errors, disrupting the coherence of action sequences. To address this, we propose a Multi-Modal Graph Convolutional Network (MMGCN) that integrates low-frame-rate (e.g., 1 fps) visual data with high-frame-rate (e.g., 30 fps) motion data (skeleton and object detections) to mitigate fragmentation. Our framework introduces three key contributions. First, a sinusoidal encoding strategy that maps 3D skeleton coordinates into a continuous sin-cos space to enhance spatial representation robustness. Second, a temporal graph fusion module that aligns multi-modal inputs with differing resolutions via hierarchical feature aggregation, Third, inspired by the smooth transitions inherent to human actions, we design SmoothLabelMix, a data augmentation technique that mixes input sequences and labels to generate synthetic training examples with gradual action transitions, enhancing temporal consistency in predictions and reducing over-segmentation artifacts. Extensive experiments on the Bimanual Actions Dataset, a public benchmark for human-object interaction understanding, demonstrate that our approach outperforms state-of-the-art methods, especially in action segmentation accuracy, achieving F1@10: 94.5% and F1@25: 92.8%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。