通过语音与手势的交互图模型,实现对失语症严重程度的自动评估。
Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation
- 构建多模态图模型,将词汇和手势作为节点,互动关系作为边。
- 图神经网络学习整体结构特征,准确预测失语症严重程度。
- 适合语言病理学家、康复医学及远程医疗场景使用。
失语症是由负责语言的大脑区域损伤引起的获得性语言障碍,影响书面和口语的理解与表达。西方失语症电池修订版(WAB-R)是言语语言病理学家用于评估失语症类型和严重程度的工具。由于WAB-R仅测量孤立的语言技能,近年来研究者越来越关注以话语生成作为日常语言能力的整体表征。当前语音分析进展主要依赖孤立的语言或声学特征,进行失语症严重程度的自动化估计。本文提出一种基于图神经网络的框架,将每位参与者的语言表达表示为有向多模态图,其中节点代表词汇和手势,边编码词-词、手势-词及词-手势间的转换关系。采用GraphSAGE学习个体层面的嵌入表示,整合邻域信息与全局图结构。结果表明,失语症严重程度并非编码于孤立的词汇分布中,而是源于语音与手势之间的结构化互动。所提架构可实现可靠的自动化失语症评估,适用于床旁筛查和基于远程医疗的监测。
原文摘要 · Abstract (English)
Aphasia is an acquired language disorder caused by injury to the regions of the brain that are responsible for language. Aphasia may impair the use and comprehension of written and spoken language. The Western Aphasia Battery-Revised (WAB-R) is an assessment tool administered by speech-language pathologists (SLPs) to evaluate the aphasia type and severity. Because the WAB-R measures isolated linguistic skills, there has been growing interest in the assessment of discourse production as a more holistic representation of everyday language abilities. Recent advancements in speech analysis focus on automated estimation of aphasia severity from spontaneous speech, relying mostly in isolated linguistic or acoustical features. In this work, we propose a graph neural network-based framework for estimating aphasia severity. We represented each participant's discourse as a directed multi-modal graph, where nodes represent lexical items and gestures and edges encode word-word, gesture-word, and word-gesture transitions. GraphSAGE is employed to learn participant-level embeddings, thus integrating information from immediate neighbors and overall graph structure. Our results suggest that aphasia severity is not encoded in isolated lexical distribution, but rather emerges from structured interactions between speech and gesture. The proposed architecture offers a reliable automated aphasia assessment, with possible uses in bedside screening and telehealth-based monitoring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。