融合视觉与深度信息，用大模型解释帕金森步态异常。
Explainable Parkinsons Disease Gait Recognition Using Multimodal RGB-D Fusion and Large Language Models
- 双分支YOLOv11提取RGB与深度特征，融合多尺度时空信息。
- 在低光、遮挡等条件下准确识别步态异常，精度优于单模态方法。
- 大模型生成临床可读报告，适合医疗辅助诊断场景。
精准且可解释的步态分析对帕金森病（PD）的早期检测至关重要，但现有方法受限于单模态输入、鲁棒性差及临床透明度不足。本文提出一种可解释的多模态框架，整合RGB与深度（RGB-D）数据，在真实环境下识别帕金森步态模式。系统采用双支路YOLOv11编码器进行模态特异性特征提取，结合多尺度局部-全局提取（MLGE）模块与跨空间颈部融合机制，增强时空表征能力，有效捕捉细微肢体运动（如摆臂减少）与整体步态动态（如步幅缩短或转身困难），即使在低光照或衣物遮挡等挑战场景下亦表现稳定。为确保可解释性，引入冻结的大语言模型（LLM），将融合后的视觉嵌入与结构化元数据转换为临床有意义的文本说明。在多模态步态数据集上的实验表明，该框架在识别精度、环境变化鲁棒性以及视觉-语言推理清晰度方面均优于单输入基线方法。通过融合多模态特征学习与语言解释能力，本研究弥合了视觉识别与临床理解之间的鸿沟，为可靠且可解释的帕金森病步态分析提供了新颖的视觉-语言范式。
原文摘要 · Abstract (English)
Accurate and interpretable gait analysis plays a crucial role in the early detection of Parkinsons disease (PD),yet most existing approaches remain limited by single-modality inputs, low robustness, and a lack of clinical transparency. This paper presents an explainable multimodal framework that integrates RGB and Depth (RGB-D) data to recognize Parkinsonian gait patterns under realistic conditions. The proposed system employs dual YOLOv11-based encoders for modality-specific feature extraction, followed by a Multi-Scale Local-Global Extraction (MLGE) module and a Cross-Spatial Neck Fusion mechanism to enhance spatial-temporal representation. This design captures both fine-grained limb motion (e.g., reduced arm swing) and overall gait dynamics (e.g., short stride or turning difficulty), even in challenging scenarios such as low lighting or occlusion caused by clothing. To ensure interpretability, a frozen Large Language Model (LLM) is incorporated to translate fused visual embeddings and structured metadata into clinically meaningful textual explanations. Experimental evaluations on multimodal gait datasets demonstrate that the proposed RGB-D fusion framework achieves higher recognition accuracy, improved robustness to environmental variations, and clear visual-linguistic reasoning compared with single-input baselines. By combining multimodal feature learning with language-based interpretability, this study bridges the gap between visual recognition and clinical understanding, offering a novel vision-language paradigm for reliable and explainable Parkinsons disease gait analysis. Code:https://github.com/manaralnaasan/RGB-D_parkinson-LLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。