用大模型实现音频与动作数据的晚融合，零样本识别活动
Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition
- 用大模型对音视频、运动数据做晚融合，不需特定训练
- 零样本和一样本下12类活动识别F1均显著高于随机水平
- 适合数据少、难对齐的多模态时序应用，部署成本低
传感器数据流为下游应用提供了丰富的活动与上下文信息，但整合互补信息仍具挑战。本文展示大语言模型（LLMs）可用于从音频和运动时间序列数据中进行晚融合以实现活动分类。我们从Ego4D数据集筛选出涵盖家庭活动、体育等多样场景的子集用于活动识别。评估结果显示，使用LLM进行融合的零样本和一样本分类在12类活动中取得显著高于随机水平的F1分数，且无需任务特定训练。基于LLM的融合方法可在缺乏对齐训练数据的情况下，实现多模态时序应用的零样本分类；同时，该方法无需为特定应用额外部署多模态模型，节省内存与计算开销。
原文摘要 · Abstract (English)
Sensor data streams provide valuable information around activities and context for downstream applications, though integrating complementary information can be challenging. We show that large language models (LLMs) can be used for late fusion for activity classification from audio and motion time series data. We curated a subset of data for diverse activity recognition across contexts (e.g., household activities, sports) from the Ego4D dataset. Evaluated LLMs achieved 12-class zero- and one-shot classification F1-scores significantly above chance, with no task-specific training. Zero-shot classification via LLM-based fusion from modality-specific models can enable multimodal temporal applications where there is limited aligned training data for learning a shared embedding space. Additionally, LLM-based fusion can enable model deploying without requiring additional memory and computation for targeted application-specific multimodal models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。