一个模型同时预测多人社交信号，提升对话理解精度。
M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention
- 设计块级注意力机制，同步处理多模态社交信号。
- 在HHCD数据集上，多模态融合使咬食时间预测准确率提升。
- 适合人机交互与社交智能研究者参考。
理解多人对话中的社交信号对人机交互和人工社会智能至关重要。社交信号包括身体姿态、头部朝向、语音及特定情境行为（如用餐时取食和咀嚼）。以往研究通常为每项任务构建专用模型。本文提出M3PT，一种因果变压器架构，通过模态与时间块级注意力掩码，实现对多名参与者及其时序互动中多种社交线索的联合建模。在人类共餐数据集（HHCD）上训练与评估，结果表明多模态输入显著提升了咬食时间与说话状态的预测性能。代码已开源。
原文摘要 · Abstract (English)
Understanding social signals in multi-party conversations is important for human-robot interaction and artificial social intelligence. Social signals include body pose, head pose, speech, and context-specific activities like acquiring and taking bites of food when dining. Past work in multi-party interaction tends to build task-specific models for predicting social signals. In this work, we address the challenge of predicting multimodal social signals in multi-party settings in a single model. We introduce M3PT, a causal transformer architecture with modality and temporal blockwise attention masking to simultaneously process multiple social cues across multiple participants and their temporal interactions. We train and evaluate M3PT on the Human-Human Commensality Dataset (HHCD), and demonstrate that using multiple modalities improves bite timing and speaking status prediction. Source code: https://github.com/AbrarAnwar/masked-social-signals/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。