arXiv:2511.16991cs.CV2025-11

仅用视觉特征融合自监督与卷积模型,精准预测图像复杂度。

DReX: Pure Vision Fusion of Self-Supervised and Convolutional Representations for Image Complexity Prediction

  • 用可学习注意力融合DINOv3与ResNet-50的多尺度特征。
  • 在IC9600上皮尔逊相关系数达0.9581,参数量少21.5倍。
  • 无需语言信息,适合图像压缩与认知科学应用。

视觉复杂度预测是计算机视觉的基础问题,应用于图像压缩、检索和分类。理解人类如何感知图像复杂度也是认知科学长期关注的问题。现有方法多依赖图文多模态模型,但语言信息是否必要尚不明确。本文提出DReX(DINO-ResNet Fusion),一个仅使用视觉信息的模型,通过可学习注意力机制融合自监督与卷积特征,以预测图像复杂度。该架构结合了ResNet-50的多尺度层次特征与DINOv3 ViT-S/16的语义丰富表示,能够同时捕捉低层纹理模式与高层语义结构。DReX在IC9600基准上取得当前最佳性能(皮尔逊相关系数r = 0.9581),超越此前使用图文数据训练的方法,且参数量仅约为其1/21.5。此外,DReX在多个数据集与指标上表现稳健,在皮尔逊与斯皮尔曼相关性、均方根误差(RMSE)及平均绝对误差(MAE)方面均优于现有方法。消融实验与注意力分析表明,DReX有效利用双骨干网络的互补信息,其中DINOv3 [CLS] token增强了对视觉复杂度的敏感性。研究结果表明,仅靠视觉特征即可实现与人类感知对齐的复杂度预测,且当合理融合时,自监督视觉变压器与监督卷积神经网络在该任务中具有互补与协同优势。

原文摘要 · Abstract (English)

Visual complexity prediction is a fundamental problem in computer vision with applications in image compression, retrieval, and classification. Understanding what makes humans perceive an image as complex is also a long-standing question in cognitive science. Recent approaches have leveraged multimodal models that combine visual and linguistic representations, but it remains unclear whether language information is necessary for this task. We propose DReX (DINO-ResNet Fusion), a vision-only model that fuses self-supervised and convolutional representations through a learnable attention mechanism to predict image complexity. Our architecture integrates multi-scale hierarchical features from ResNet-50 with semantically rich representations from DINOv3 ViT-S/16, enabling the model to capture both low-level texture patterns and high-level semantic structure. DReX achieves state-of-the-art performance on the IC9600 benchmark (Pearson r = 0.9581), surpassing previous methods--including those trained on multimodal image-text data--while using approximately 21.5x fewer learnable parameters. Furthermore, DReX generalizes robustly across multiple datasets and metrics, achieving superior results on Pearson and Spearman correlation, Root Mean Square Error (RMSE), and Mean Absolute Error (MAE). Ablation and attention analyses confirm that DReX leverages complementary cues from both backbones, with the DINOv3 [CLS] token enhancing sensitivity to visual complexity. Our findings suggest that visual features alone can be sufficient for human-aligned complexity prediction and that, when properly fused, self-supervised transformers and supervised deep convolutional neural networks offer complementary and synergistic benefits for this task.

图像复杂度视觉融合自监督学习深度模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。