arXiv:2505.02052cs.AIcs.CV2025-05被引 5

用文本和压力数据互生成,提升动作识别准确率

TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity Recognition

  • 双向生成:文本转压力、压力转文本,利用大模型理解动态
  • 在瑜伽等真实任务中提升12.4%宏平均F1分数
  • 适合做压力传感器动作识别、数据增强的研究者

基于传感器的人类活动识别(HAR)长期依赖惯性测量单元和视觉数据,忽视了压力传感器在捕捉细微身体动态与重心变化方面的独特优势。尽管其对姿势与平衡相关活动有潜力,但因数据集有限,应用仍受限制。为此,我们提出结合生成式基础模型与专用于压力传感器的HAR技术。具体地,设计了双向文本×压力模型TxP,利用CLIP和LLaMA 2 13B Chat等预训练模型,将压力数据转化为自然语言,并实现两大任务:(1) 文本到压力(Text2Pressure),将活动描述生成压力序列;(2) 压力到文本(Pressure2Text),从动态压力图生成活动描述与分类。模型在包含81,100个文本-压力对的合成数据集PressLang上训练,经真实世界数据验证,涵盖瑜伽与日常任务,实现了基于原子动作的数据增强与分类,相较当前最优方法,在宏平均F1分数上最高提升12.4%,推动压力驱动的HAR向更广泛应用与深层运动理解迈进。

原文摘要 · Abstract (English)

Sensor-based human activity recognition (HAR) has predominantly focused on Inertial Measurement Units and vision data, often overlooking the capabilities unique to pressure sensors, which capture subtle body dynamics and shifts in the center of mass. Despite their potential for postural and balance-based activities, pressure sensors remain underutilized in the HAR domain due to limited datasets. To bridge this gap, we propose to exploit generative foundation models with pressure-specific HAR techniques. Specifically, we present a bidirectional Text$\times$Pressure model that uses generative foundation models to interpret pressure data as natural language. TxP accomplishes two tasks: (1) Text2Pressure, converting activity text descriptions into pressure sequences, and (2) Pressure2Text, generating activity descriptions and classifications from dynamic pressure maps. Leveraging pre-trained models like CLIP and LLaMA 2 13B Chat, TxP is trained on our synthetic PressLang dataset, containing over 81,100 text-pressure pairs. Validated on real-world data for activities such as yoga and daily tasks, TxP provides novel approaches to data augmentation and classification grounded in atomic actions. This consequently improved HAR performance by up to 12.4\% in macro F1 score compared to the state-of-the-art, advancing pressure-based HAR with broader applications and deeper insights into human movement.

动作识别压力传感生成模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。