用大模型实现空中写字识别，准确率提升129倍
Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understanding
- 用自建数据微调+少样本学习，提升大模型对二维写字的识别能力
- 设计编码器将三维轨迹转为二维表示，保留时空特征用于写字预测
- 在空中写5个字母以内单词，识别准确率达78%，适合动作精细识别场景
基于惯性测量单元（IMU）的人体活动识别（HAR）越来越多地采用大语言模型（LLMs），但现有方法主要关注步行、跑步等粗粒度活动。我们初步研究表明，预训练大模型在细粒度任务（如空中写字识别）上表现极差，准确率仅接近随机猜测。本文首先解决平面书写场景：通过微调自建数据集并结合少样本学习，二维数据上的识别准确率最高提升129倍。为扩展至三维场景，设计了一个编码器管道，将3D数据映射为2D等效表示，有效保留时空信息以支持鲁棒写字识别。最终端到端系统在空中写5个字母以内的单词任务中达到78%的准确率，证明了大模型在细粒度人体活动理解中的可行性。
原文摘要 · Abstract (English)
Human activity recognition (HAR) using inertial measurement units (IMUs) increasingly leverages large language models (LLMs), yet existing approaches focus on coarse activities like walking or running. Our preliminary study indicates that pretrained LLMs fail catastrophically on fine-grained HAR tasks such as air-written letter recognition, achieving only near-random guessing accuracy. In this work, we first bridge this gap for flat-surface writing scenarios: by fine-tuning LLMs with a self-collected dataset and few-shot learning, we achieved up to a 129x improvement on 2D data. To extend this to 3D scenarios, we designed an encoder-based pipeline that maps 3D data into 2D equivalents, preserving the spatiotemporal information for robust letter prediction. Our end-to-end pipeline achieves 78% accuracy on word recognition with up to 5 letters in mid-air writing scenarios, establishing LLMs as viable tools for fine-grained HAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。