用大模型+增强现实，实时帮用户社交破冰
SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions
- 通过多模态感知提取语音、表情等社交线索，结合大模型生成建议
- 实测互动参与度比基线高38.3%，95%用户愿在真实社交中使用
- 适合需要提升社交能力或焦虑人群，尤其适用于实时互动场景
社交是人类生活的核心。近年来，基于大语言模型（LLMs）的虚拟助手展现出重塑人际互动与生活方式的巨大潜力。然而，现有辅助系统多为被动响应个体用户需求，难以在实时社交对话中提供即时协助。本研究提出SocialMind，首个基于大模型的主动式增强现实（AR）社交辅助系统，可为用户提供现场社交支持。SocialMind采用类人感知机制，利用多模态传感器捕捉言语与非言语线索、社交因素及隐含人格特征，并将这些社交信号融入大模型推理以生成社交建议。同时，系统采用多层协同生成策略与主动更新机制，将建议实时投射至增强现实眼镜上，确保建议及时呈现且不打断自然对话流程。在三个公开数据集及20名用户的实验中，SocialMind相较基线实现38.3%更高的互动参与度，95%参与者表示愿意在真实社交中使用该系统。
原文摘要 · Abstract (English)
Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。