arXiv:2411.00813cs.MMcs.AI2024-11被引 2

通过多模态对齐与域适应,提升短视频中人格分析的准确性与泛化能力。

Personality Analysis from Online Short Video Platforms with Multi-domain Adaptation

  • 基于语音时间戳对齐多模态数据,实现跨模态精准同步。
  • 融合双向LSTM与自注意力机制,有效捕捉行为时序特征。
  • 采用梯度域适应方法,在少量标注数据下仍保持良好性能。

在线短视频中的人格分析因个性化推荐、情感分析及人机交互等应用而日益重要。传统基于大五人格框架的问卷评估受自我报告偏差影响,难以大规模或实时应用。利用短视频中的丰富多模态数据可为更准确的人格推断提供新路径,但如何对齐异步、多样化的模态数据并提升模型在新领域的泛化能力仍是挑战。本文提出一种新型多模态人格分析框架,通过基于时间戳的模态对齐机制实现跨模态数据同步,确保特征整合精度;采用双向长短期记忆网络与自注意力机制建模时序依赖和模态间交互,聚焦关键信息;设计一种基于梯度的域适应方法,将多个源域知识迁移至标签稀缺的目标域。在真实数据集上的大量实验表明,该框架在人格预测任务上显著优于现有方法,展现出对复杂行为线索的捕捉能力与强域适应性。

原文摘要 · Abstract (English)

Personality analysis from online short videos has gained prominence due to its applications in personalized recommendation systems, sentiment analysis, and human-computer interaction. Traditional assessment methods, such as questionnaires based on the Big Five Personality Framework, are limited by self-report biases and are impractical for large-scale or real-time analysis. Leveraging the rich, multi-modal data present in short videos offers a promising alternative for more accurate personality inference. However, integrating these diverse and asynchronous modalities poses significant challenges, particularly in aligning time-varying data and ensuring models generalize well to new domains with limited labeled data. In this paper, we propose a novel multi-modal personality analysis framework that addresses these challenges by synchronizing and integrating features from multiple modalities and enhancing model generalization through domain adaptation. We introduce a timestamp-based modality alignment mechanism that synchronizes data based on spoken word timestamps, ensuring accurate correspondence across modalities and facilitating effective feature integration. To capture temporal dependencies and inter-modal interactions, we employ Bidirectional Long Short-Term Memory networks and self-attention mechanisms, allowing the model to focus on the most informative features for personality prediction. Furthermore, we develop a gradient-based domain adaptation method that transfers knowledge from multiple source domains to improve performance in target domains with scarce labeled data. Extensive experiments on real-world datasets demonstrate that our framework significantly outperforms existing methods in personality prediction tasks, highlighting its effectiveness in capturing complex behavioral cues and robustness in adapting to new domains.

人格分析多模态域适应短视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。