通过同步视觉与面部特征互提,提升犹豫与矛盾行为识别准确率。
SVF-CR: Synchronized Visual-Facial Cross-Refinement for Multimodal Ambivalence and Hesitancy Recognition

- 用同步分割的视频与人脸片段,双向交叉精炼特征。
- 在公开数据集上实现0.7156的宏平均F1,优于现有方法。
- 适合多模态行为分析、人机交互等场景研究者参考。
犹豫与矛盾是通过语言内容、面部表情、视觉上下文和声学线索共同表达的细微行为状态。有效识别不仅需提取各模态的有用表征,还需建模跨模态时间对齐行为证据间的交互关系。本文提出一种同步视觉-面部交叉精炼框架(SVF-CR),采用成对多模态证据融合来识别此类行为。该方法首先使用相同时间划分提取全视频段令牌与裁剪人脸段令牌;同步的视觉与面部令牌通过模内自注意力及双向视觉-面部交叉注意力进行精炼,使整体视频上下文与局部面部行为相互增强。随后,通过一致性与差异性建模构建段级视觉-面部证据,并经时序自注意力与注意力池化处理。文本与声学特征通过上下文自注意力轻度精炼,最终在决策阶段以成对证据融合方式与强化后的视觉-面部证据结合。在BAH公开评测集上的实验表明,所提方法在公共宏平均F1上超越全局视觉-面部令牌融合与同步证据基线,达到0.7156。代码已开源:https://github.com/hiinnnii/BAH-Challenge-ECCV2026_SVF-CR。
原文摘要 · Abstract (English)
Ambivalence and hesitancy are subtle behavioral states that are expressed through a combination of verbal content, facial behavior, visual context, and acoustic cues. Effective recognition therefore requires not only extracting informative unimodal representations, but also modeling how temporally aligned behavioral evidence interacts across modalities. In this paper, we propose a synchronized visual-facial cross-refinement framework (SVF-CR) with pairwise multimodal evidence fusion for ambivalence and hesitancy recognition. The proposed method first extracts whole-video segment tokens and cropped-face segment tokens using the same temporal partition. The synchronized visual and facial tokens are refined through intra-modal self-attention and bidirectional visual-facial cross-attention, allowing whole-video context and local facial behavior to mutually refine each other before evidence construction. We then construct segment-level visual-facial evidence using consistency and discrepancy modeling, followed by temporal self-attention and attention pooling. Textual and acoustic features are lightly refined through context self-attention and are fused with the enhanced visual-facial evidence at the final decision stage using pairwise evidence fusion. Experiments on the BAH (Behavioral Ambivalence/Hesitancy) public evaluation split show that the proposed synchronized visual-facial cross-refinement improves public macro-F1 over both global visual-face token fusion and synchronized evidence baselines, achieving a public macro-F1 of 0.7156. Code is available at : https://github.com/hiinnnii/BAH-Challenge-ECCV2026\_SVF-CR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。