评测大模型对传统中国画的评析能力，发现其在文化语境理解上的短板。
A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks
- 构建多维度评画框架，量化专家评论中的立场、焦点与质量
- 通过角色引导提示，测试大模型从不同视角生成评述的能力
- 揭示当前模型在文化情境理解中的不足，适合艺术与AI交叉研究者参考
本研究旨在评估主流视觉语言模型（VLMs）在生成中国传统绘画评论方面的能力。为此，我们首先开发了一个定量评价框架，通过零样本分类模型从人类专家评论中提取涵盖评价立场、特征关注点和评论质量的多维评估特征，并据此定义和量化多个代表性评论角色。该框架被用于评估Llama、Qwen、Gemini等典型VLMs。实验采用角色引导提示策略，考察模型生成多样化评论的能力。结果揭示了当前VLMs在艺术评析任务中的表现水平、优势与改进空间，为理解其在复杂语义理解与内容生成中的潜力与局限提供了洞见。实验代码已公开：https://github.com/yha9806/VULCA-EMNLP2025。
原文摘要 · Abstract (English)
This study aims to test and evaluate the capabilities and characteristics of current mainstream Visual Language Models (VLMs) in generating critiques for traditional Chinese painting. To achieve this, we first developed a quantitative framework for Chinese painting critique. This framework was constructed by extracting multi-dimensional evaluative features covering evaluative stance, feature focus, and commentary quality from human expert critiques using a zero-shot classification model. Based on these features, several representative critic personas were defined and quantified. This framework was then employed to evaluate selected VLMs such as Llama, Qwen, or Gemini. The experimental design involved persona-guided prompting to assess the VLM's ability to generate critiques from diverse perspectives. Our findings reveal the current performance levels, strengths, and areas for improvement of VLMs in the domain of art critique, offering insights into their potential and limitations in complex semantic understanding and content generation tasks. The code used for our experiments can be publicly accessed at: https://github.com/yha9806/VULCA-EMNLP2025.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。