用图结构建模多模态互动，提升短视频人格预测准确率
GAME: Learning Multimodal Interactions via Graph Structures for Personality Trait Estimation
- 构建面部图结构，融合图卷积与视觉特征捕捉表情与身份信息
- 在3个基准数据集上超越现有方法，最高提升6.2%的预测准确率
- 适合研究多模态融合、人格分析或视频理解的开发者参考
从短视频中进行人格分析面临视觉、听觉和文本线索复杂交互的挑战。本文提出GAME——一种基于图结构的多模态编码器，用于鲁棒地建模与融合多源特征以实现自动人格预测。视觉流中，构建面部图并引入双分支Geo Two-Stream网络，结合图卷积网络（GCNs）与卷积神经网络（CNNs）及注意力机制，捕捉面部结构与外观特征；同时使用预训练的ResNet18和VGGFace提取全局上下文与身份特征。为捕捉时间动态，帧级特征通过引入时序注意力模块的BiGRU处理。音频表示来自VGGish网络，语言语义由XLM-Roberta Transformer捕获。为实现有效多模态融合，提出基于通道注意力的融合模块，后接多层感知机（MLP）回归头进行人格特质预测。大量实验表明，GAME在多个基准数据集上持续优于现有方法，验证了其有效性和泛化能力。
原文摘要 · Abstract (English)
Apparent personality analysis from short videos poses significant chal-lenges due to the complex interplay of visual, auditory, and textual cues. In this paper, we propose GAME, a Graph-Augmented Multimodal Encoder designed to robustly model and fuse multi-source features for automatic personality prediction. For the visual stream, we construct a facial graph and introduce a dual-branch Geo Two-Stream Network, which combines Graph Convolutional Networks (GCNs) and Convolutional Neural Net-works (CNNs) with attention mechanisms to capture both structural and appearance-based facial cues. Complementing this, global context and iden-tity features are extracted using pretrained ResNet18 and VGGFace back-bones. To capture temporal dynamics, frame-level features are processed by a BiGRU enhanced with temporal attention modules. Meanwhile, audio representations are derived from the VGGish network, and linguistic se-mantics are captured via the XLM-Roberta transformer. To achieve effective multimodal integration, we propose a Channel Attention-based Fusion module, followed by a Multi-Layer Perceptron (MLP) regression head for predicting personality traits. Extensive experiments show that GAME con-sistently outperforms existing methods across multiple benchmarks, vali-dating its effectiveness and generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。