用混合模型捕捉人脸局部与整体特征关系,提升颜值预测准确率
Scale-interaction transformer: a hybrid cnn-transformer model for facial beauty prediction
- 结合CNN多尺度提取与Transformer建模跨尺度关联
- 在SCUT-FBP5500上达到0.9187的皮尔逊相关系数
- 适合需要全局上下文理解的图像回归任务
自动化人脸颜值预测(FBP)是计算机视觉中一项挑战性任务,因局部与全局面部特征之间的复杂交互影响人类感知。虽然卷积神经网络(CNN)擅长特征提取,但通常以固定尺度处理信息,可能忽略不同粒度特征间的关键依赖关系。为此,我们提出一种新型混合深度学习架构——尺度交互变压器(Scale-Interaction Transformer, SIT),融合了CNN的特征提取能力与Transformer的关系建模优势。SIT首先通过并行卷积的多尺度模块捕获不同感受野下的面部特征;随后将这些多尺度表示构造成序列,由Transformer编码器利用自注意力机制显式建模其交互与上下文关系。我们在广泛使用的SCUT-FBP5500基准数据集上进行了大量实验,所提出的SIT模型取得了新的最优性能,皮尔逊相关系数达0.9187,优于以往方法。结果表明,显式建模多尺度视觉线索间的相互作用对高性能颜值预测至关重要。SIT架构的成功凸显了混合CNN-Transformer模型在需整体、上下文感知理解的复杂图像回归任务中的潜力。
原文摘要 · Abstract (English)
Automated Facial Beauty Prediction (FBP) is a challenging computer vision task due to the complex interplay of local and global facial features that influence human perception. While Convolutional Neural Networks (CNNs) excel at feature extraction, they often process information at a fixed scale, potentially overlooking the critical inter-dependencies between features at different levels of granularity. To address this limitation, we introduce the Scale-Interaction Transformer (SIT), a novel hybrid deep learning architecture that synergizes the feature extraction power of CNNs with the relational modeling capabilities of Transformers. The SIT first employs a multi-scale module with parallel convolutions to capture facial characteristics at varying receptive fields. These multi-scale representations are then framed as a sequence and processed by a Transformer encoder, which explicitly models their interactions and contextual relationships via a self-attention mechanism. We conduct extensive experiments on the widely-used SCUT-FBP5500 benchmark dataset, where the proposed SIT model establishes a new state-of-the-art. It achieves a Pearson Correlation of 0.9187, outperforming previous methods. Our findings demonstrate that explicitly modeling the interplay between multi-scale visual cues is crucial for high-performance FBP. The success of the SIT architecture highlights the potential of hybrid CNN-Transformer models for complex image regression tasks that demand a holistic, context-aware understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。