用大模型生成伪标签,提升视频隐性情绪识别准确率
Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks
- 用Gemini生成伪标签和推理文本,实现多模态弱监督训练
- 在iMiGUE数据集上准确率突破0.69,显著优于之前方法
- 轻量MLP替代图网络,高效建模关键点时序关系
为解决视频中隐蔽情绪的自动识别问题,本文提出一种多模态弱监督框架,在iMiGUE网球访谈数据集上取得当前最优性能。首先,YOLO 11x逐帧检测并裁剪人脸,DINOv2-Base提取视觉特征;随后,通过链式思维与反思提示(CoT + Reflection),Gemini 2.5 Pro自动生成伪标签和推理文本,作为下游模型的弱监督信号;接着,OpenPose生成137维关键点序列,并融合帧间偏移特征,将传统图神经网络简化为MLP以高效建模时空关系;超长序列Transformer独立编码图像与关键点序列,其表示与BERT编码的访谈文本拼接后联合微调。各模态先独立预训练,再联合微调,伪标签样本加入训练集进一步提升性能。实验表明,尽管存在严重类别不平衡,该方法将准确率从此前不足0.6提升至超过0.69,建立新基准。研究还验证了'MLP化'的关键点骨干可媲美甚至超越基于GCN的方案。
原文摘要 · Abstract (English)
To tackle the automatic recognition of "concealed emotions" in videos, this paper proposes a multimodal weak-supervision framework and achieves state-of-the-art results on the iMiGUE tennis-interview dataset. First, YOLO 11x detects and crops human portraits frame-by-frame, and DINOv2-Base extracts visual features from the cropped regions. Next, by integrating Chain-of-Thought and Reflection prompting (CoT + Reflection), Gemini 2.5 Pro automatically generates pseudo-labels and reasoning texts that serve as weak supervision for downstream models. Subsequently, OpenPose produces 137-dimensional key-point sequences, augmented with inter-frame offset features; the usual graph neural network backbone is simplified to an MLP to efficiently model the spatiotemporal relationships of the three key-point streams. An ultra-long-sequence Transformer independently encodes both the image and key-point sequences, and their representations are concatenated with BERT-encoded interview transcripts. Each modality is first pre-trained in isolation, then fine-tuned jointly, with pseudo-labeled samples merged into the training set for further gains. Experiments demonstrate that, despite severe class imbalance, the proposed approach lifts accuracy from under 0.6 in prior work to over 0.69, establishing a new public benchmark. The study also validates that an "MLP-ified" key-point backbone can match - or even surpass - GCN-based counterparts in this task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。