用冻结的多模态模型从视频面试中预测性格与认知能力，效果优于基线。
Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability
- 采用冻结的CLIP、Whisper、RoBERTa等模型提取视觉、音频和文本特征
- 性格预测平均MSE降至0.2696，比基线降低19.1%相对误差
- 发现认知能力预测可能存在数据集捷径，需谨慎评估模型可靠性
从异步视频面试（AVI）中预测心理特质是人工智能辅助面试中的挑战性问题，因标注数据有限且每段回答包含高维视觉、声学和语言信号。本文针对ACM多媒体AVI挑战赛2026提出解决方案，涵盖两个任务：Track~1基于性格相关回答预测自报的HEXACO人格特质；Track~2对结构化回答进行认知能力等级分类。我们将其视为小样本表示学习任务，不微调大模型，而是使用冻结的多模态编码器（包括用于视觉的CLIP、用于声学与转录的Whisper，以及用于文本的RoBERTa、E5和DeBERTaV3），搭配低容量下游模型。在Track~1中，特质量化回归与后期融合系统在验证集上达到平均MSE为0.2696，优于官方基线0.3334。消融实验显示：从全局模型（0.3189）到逐特质建模（0.2871）再到逐特质后期融合（0.2696），相对基线减少19.1%误差。在Track~2中，紧凑的主体-属性基线达0.5781准确率，多模态集成达0.5313，均高于官方基线0.4062。我们解释该结果可能源于验证集中的主体-属性捷径，而非真正从视频内容推断认知能力。总体表明，心理评估需依赖特质特定的多模态建模，而认知能力预测需防范数据集偏差。
原文摘要 · Abstract (English)
Predicting psychological traits from asynchronous video interviews (AVIs) is a challenging problem in AI-assisted interview assessment because labeled datasets are limited while each response contains high-dimensional visual, acoustic, and verbal signals. This paper presents our solution for the ACM Multimedia AVI Challenge 2026, which evaluates two tasks: Track~1 predicts self-reported HEXACO personality traits from personality-related interview responses, and Track~2 classifies cognitive ability levels from structured AVI responses. We treat the problem as a small-sample representation learning task. Instead of fine-tuning large pretrained models, we use frozen multimodal encoders, including CLIP for visual features, Whisper for acoustic features and transcripts, and RoBERTa, E5, and DeBERTaV3 for textual representations, followed by low-capacity downstream models. For Track~1, our trait-specific regression and late-fusion system achieves an average validation MSE of 0.2696, improving over the official baseline of 0.3334. Ablation results show a three-step improvement from a global model (0.3189), to per-trait modeling (0.2871), to per-trait late fusion (0.2696), corresponding to a 19.1% relative MSE reduction over the official baseline. For Track~2, a compact subject-attribute baseline reaches 0.5781 accuracy, while our multimodal ensemble reaches 0.5313, both above the official baseline of 0.4062. We interpret this result as evidence of possible subject-attribute shortcuts in the validation split rather than robust cognitive inference from AVI content. Overall, our findings suggest that AVI-based psychological assessment benefits from trait-specific multimodal modeling, but cognitive ability prediction requires careful control of dataset shortcuts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。