提出新模型识别真实治疗场景中的松散互动,助力自闭症社交能力评估。
Loose Social-Interaction Recognition in Real-world Therapy Scenarios
- 双路架构结合全局注意力机制捕捉非同步双人互动
- 在自闭症数据集上达到当前最佳性能,验证有效性
- 揭示不同社交行为需定制化网络设计,适配医疗评估场景
计算机视觉领域已研究如推、搬运等基础双人动作,但随着深度学习发展,亟需探索更复杂的松散互动——即两人各自执行原子动作以完成整体任务,且无需时间同步或物理接触(如共同做饭)。这类互动分析对心理疾病诊断与社交技能训练具有重要价值。为此,本文提出一种新型双路架构,通过CNN主干提取个体全局抽象特征,并利用基于交叉注意力的全局层注意力模块进行融合。我们在真实自闭症诊断场景下的松散互动数据集及公开自闭症数据集上评估模型,均取得基线或最优结果。此外,通过在NTU-RGB+D数据集上实验发现,不同交互类型需不同网络设计。还对比了加入时序信息的改进版本,在紧致互动任务中也达到当前最优表现。
原文摘要 · Abstract (English)
The computer vision community has explored dyadic interactions for atomic actions such as pushing, carrying-object, etc. However, with the advancement in deep learning models, there is a need to explore more complex dyadic situations such as loose interactions. These are interactions where two people perform certain atomic activities to complete a global action irrespective of temporal synchronisation and physical engagement, like cooking-together for example. Analysing these types of dyadic-interactions has several useful applications in the medical domain for social-skills development and mental health diagnosis. To achieve this, we propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module based on a cross-attention strategy. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions. Our network achieves baseline results on the Loose-Interaction and SOTA results on the Autism datasets. Moreover, we study different social interactions by experimenting on a publicly available dataset i.e. NTU-RGB+D (interactive classes from both NTU-60 and NTU-120). We have found that different interactions require different network designs. We also compare a slightly different version of our method by incorporating time information to address tight interactions achieving SOTA results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。