用对比学习让传感器数据与语义描述对齐，提升零样本动作识别准确率。
Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data
- 用文本描述替代标签名作为原型，增强传感器与语义的对齐
- 对比训练使未见动作识别准确率达73.2%，F1达0.583
- 建议用宏平均F1评估不平衡数据集，避免误判
基于惯性测量单元（IMU）的零样本人体动作识别（HAR）面临传感器嵌入与语义类别表示之间模态鸿沟的挑战。在PAMAP2数据集上，我们系统评估了七种配置，包含三种推理方法与两种训练流程，使用14个已见类和4个未见类，测试时剔除受试者108和109。发现模态鸿沟是训练阶段由编码器目标决定的现象。采用交叉熵损失训练的时序卷积网络（TCN），以标签名称为句向量原型时，传感器嵌入与对应文本原型的平均余弦相似度为0.30；改用更具区分性的动作描述后，相似度提升至0.69。该对齐改进在所有三种推理方法中均有效。最佳结果结合对比训练与反向Softmax校正，在未见类上达到73.2%准确率与0.583宏平均F1，优于标签名称基线的58.3%准确率与0.34宏平均F1。此外，更丰富的文本描述会降低句向量空间中的原型可分性，因共享生物力学词汇导致语言模型压缩原型分布。但只要保留足够区分性词汇，对比对齐仍具优势。我们还指出，当测试集类别分布不均衡时，整体准确率是误导性指标，建议将宏平均F1作为零样本HAR基准的标准报告指标。
原文摘要 · Abstract (English)
Zero-shot learning (ZSL) for inertial measurement unit (IMU)-based human activity recognition (HAR) faces a central challenge: bridging the gap between sensor embeddings and semantic class representations. We systematically evaluate seven configurations combining three inference methods with two training pipelines on the PAMAP2 dataset, using 14 seen and 4 unseen activity classes with subjects 108 and 109 held out for testing. We find that the modality gap is a training-time phenomenon governed by the encoder objective. A temporal convolutional network (TCN) trained with cross-entropy over label-name Sentence- BERT prototypes yields sensor embeddings with a mean cosine similarity of 0.30 to the corresponding text prototypes, while replacing the label-name prototype targets with discriminative activity descriptions raises this to 0.69. This alignment improvement transfers consistently across all three inference methods. The strongest result combines contrastive training with inverted softmax correction, achieving 73.2% accuracy and 0.583 macro F1 on unseen classes, compared to 58.3% accuracy and 0.34 macro F1 for the label-name baseline. A secondary finding is that richer text descriptions reduce inter-prototype separability in Sentence-BERT space, because shared biomechanical vocabulary causes the language model to compress the prototype cloud. This effect does not negate the benefits of contrastive alignment provided prototype descriptions retain sufficient discriminative vocabulary. We also demonstrate that overall accuracy is a misleading primary metric when test-set class distributions are imbalanced, and recommend macro-averaged F1 as the standard reporting metric for ZSL-HAR benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。