用步态视频和文本指导,精准识别青少年脊柱侧弯,避免辐射风险。
Text-Guided Multi-Instance Learning for Scoliosis Screening via Gait Video Analysis
- 通过动态时间规整聚类分割步态周期,捕捉关键运动阶段。
- 在Scoliosis1K数据集上准确识别边界病例,对不平衡数据表现优异。
- 融合专家文本与大模型指导,提升模型可解释性与诊断灵敏度。
早期脊柱侧弯难以察觉,尤其在青少年中,延迟诊断可能导致严重健康问题。传统基于X光的方法存在辐射风险且依赖临床经验,难以用于大规模筛查。为此,我们提出一种文本引导的多实例学习网络TG-MILNet,利用步态视频实现无创脊柱侧弯检测。为应对步态序列的时间错位问题,采用动态时间规整(DTW)聚类将视频分割为关键步态阶段;为聚焦诊断相关特征,引入跨包时序注意力(IBTA)机制以突出关键阶段;针对边界病例识别困难,设计边界感知模型(BAM)增强对细微脊柱偏移的敏感性。同时结合领域专家文本与大语言模型(LLM)提供语义指导,提升特征表示与模型可解释性。在大规模Scoliosis1K步态数据集上的实验表明,TG-MILNet达到当前最优性能,尤其在处理类别不平衡及准确识别挑战性边界病例方面表现突出。代码已公开于https://github.com/lhqqq/TG-MILNet。
原文摘要 · Abstract (English)
Early-stage scoliosis is often difficult to detect, particularly in adolescents, where delayed diagnosis can lead to serious health issues. Traditional X-ray-based methods carry radiation risks and rely heavily on clinical expertise, limiting their use in large-scale screenings. To overcome these challenges, we propose a Text-Guided Multi-Instance Learning Network (TG-MILNet) for non-invasive scoliosis detection using gait videos. To handle temporal misalignment in gait sequences, we employ Dynamic Time Warping (DTW) clustering to segment videos into key gait phases. To focus on the most relevant diagnostic features, we introduce an Inter-Bag Temporal Attention (IBTA) mechanism that highlights critical gait phases. Recognizing the difficulty in identifying borderline cases, we design a Boundary-Aware Model (BAM) to improve sensitivity to subtle spinal deviations. Additionally, we incorporate textual guidance from domain experts and large language models (LLM) to enhance feature representation and improve model interpretability. Experiments on the large-scale Scoliosis1K gait dataset show that TG-MILNet achieves state-of-the-art performance, particularly excelling in handling class imbalance and accurately detecting challenging borderline cases. The code is available at https://github.com/lhqqq/TG-MILNet
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。