arXiv:2505.18465cs.CV2025-05

用AI理解临床运动数据,让医生能问问题直接得到专业回答。

BiomechGPT: Extending Motion-Language Models to Clinical Motion Understanding

  • 将运动数据转为可读格式,与语言模型结合实现多任务分析。
  • 在750人、71小时的临床运动数据上表现良好,模型越大效果越佳。
  • 适合康复医学研究者和临床医生快速解读复杂运动数据。

无标记运动捕捉技术的进步使高质量生物力学数据日益易得,推动了对可扩展下游分析的需求。为应对这一挑战,我们构建了基于语言模型的多模态运动-语言模型,支持灵活处理多样临床问题。研究收集了750名受试者(含运动障碍者)完成常见临床评估任务的71小时生物力学数据。为扩充训练数据,设计了一种跨格式分词器,无需配对数据即可将异构运动数据编码至统一潜在空间,实现多数据集标注融合。基于这些分词表示,构建了包含运动相关问答对的多模态数据集,并训练出BiomechGPT模型。该模型在多种临床相关任务中表现优异,性能随数据集和模型规模提升而增强,为临床医生与研究人员提供了一种全新的生物力学数据交互方式,代表了面向康复的运动分析新方向。

原文摘要 · Abstract (English)

Advances in markerless motion capture are making high-quality biomechanical data increasingly accessible, creating a growing need for scalable downstream analytics. Building a bespoke pipeline for each analysis task is time-consuming, motivating models that can flexibly handle diverse clinical questions within a single framework. Recent work has shown that fine-tuning language models to accept tokenized motion as an additional modality enables descriptive captioning of movement, raising the question of whether these models are also capable of clinically relevant motion understanding, where diverse tasks and annotations provide a natural testbed. We investigate whether such a multimodal motion--language model can answer detailed, clinically meaningful questions about movement. We collected 71 hours of biomechanical data from 750 participants, many with movement impairments, performing tasks commonly used in clinical assessment. To further expand the training dataset, we designed a cross-format tokenizer that directly encodes motion data from heterogeneous formats into a shared latent space without paired data, allowing a second dataset to be incorporated and enabling pooling annotations across datasets. From these tokenized representations, we constructed a multimodal dataset of motion-related question--answer pairs and used it to train BiomechGPT, a multimodal biomechanics--language model. BiomechGPT achieves competitive performance across a range of clinically relevant tasks, with performance scaling with both dataset and model size. It offers a new way for clinicians and researchers to interact with biomechanical data and represents a promising direction for rehabilitation-focused movement analysis. Project page: https://intelligentsensingandrehabilitation.github.io/BiomechGPT/

生物力学多模态临床分析语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。