让AI理解图片给人的感觉,提升生成内容的感染力。
From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
- 构建认知属性评测基准CogIP-Bench,量化图像感知能力
- 后训练使模型对记忆点、美感等主观属性判断更贴近人类
- 成果可迁移至图像生成,提升画面吸引力与记忆度
尽管多模态大模型擅长识别图像中的物体和描述场景,但在理解图像对人类观者的情感感受方面仍显不足。这一差距在主观认知属性(如记忆性、趣味性、美学价值、情感共鸣)上尤为明显。为此,我们提出CogIP-Bench,一个全面评估多模态大模型在图像认知属性上的基准。评估显示当前模型与人类感知存在显著偏差。我们进一步证明,通过后训练可有效弥合该差距,显著提升模型对人类判断的契合度。此外,这种习得的认知对齐不仅具备预测能力,还能迁移到下游创意任务中。将经认知对齐的模型融入图像生成流程,可引导生成更具记忆性或视觉吸引力的图像。本工作提供了衡量人类似感知的基准、增强对齐的后训练方法,并展示了其在实现更以人为本的AI中的潜力。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) are adept at answering what is in an image-identifying objects and describing scenes-they often lack the ability to understand how an image feels to a human observer. This gap is most evident when considering subjective cognitive properties, such as what makes an image memorable, funny, aesthetically pleasing, or emotionally evocative. To systematically address this challenge, we introduce CogIP-Bench, a comprehensive benchmark for evaluating MLLMs on such image cognitive properties. Our evaluation reveals a significant gap: current models are poorly aligned with human perception of these nuanced properties. We then demonstrate that a post-training phase can effectively bridge this gap, significantly enhancing the model's alignment with human judgments. Furthermore, we show that this learned cognitive alignment is not merely predictive but also transferable to downstream creative tasks. By integrating our cognitively-aligned MLLM into an image generation pipeline, we can guide the synthesis process to produce images that better embody desired traits, such as being more memorable or visually appealing. Our work provides a benchmark to measure this human-like perception, a post-training pipeline to enhance it, and a demonstration that this alignment unlocks more human-centric AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。