arXiv:2507.21917cs.CV2025-07被引 9

仅用图像分析艺术作品,结合知识检索实现精准风格识别与解释。

ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

  • 通过图像输入+知识检索,实现无需外部链接的艺术理解。
  • 风格分类F1提升8.4%,标题生成BLEU@1提升7.1%。
  • 适合艺术研究、数字博物馆及需要背景知识的多模态任务。

分析数字化艺术品面临独特挑战,需融合视觉理解与丰富的艺术、历史与上下文知识。我们提出ArtSeek,一种基于多模态大语言模型与检索增强生成的框架。该方法仅依赖图像输入,适用于无维基数据或维基百科链接的多数数字化藏品。ArtSeek包含三个核心组件:基于晚期交互检索的智能多模态检索模块、用于预测艺术家、流派、风格、媒介和标签的对比多任务分类网络,以及通过Qwen2.5-VL支持的上下文示例驱动的代理式推理策略。关键在于WikiFragments——一个大规模图像-文本片段数据集,专为知识引导的多模态推理设计。在多个基准测试中,ArtSeek达到顶尖性能:风格分类比GraphCLIP提升8.4% F1,ArtPedia上的标题生成提升7.1% BLEU@1。定性分析表明,ArtSeek可识别视觉母题、推断历史背景并检索相关知识,即使对冷门作品亦有效。尽管聚焦视觉艺术,该方法可泛化至其他需外部知识的领域,推动可扩展的多模态人工智能研究。数据集与源代码将公开于https://github.com/cilabuniba/artseek。

原文摘要 · Abstract (English)

Analyzing digitized artworks presents unique challenges, requiring not only visual interpretation but also a deep understanding of rich artistic, contextual, and historical knowledge. We introduce ArtSeek, a multimodal framework for art analysis that combines multimodal large language models with retrieval-augmented generation. Unlike prior work, our pipeline relies only on image input, enabling applicability to artworks without links to Wikidata or Wikipedia-common in most digitized collections. ArtSeek integrates three key components: an intelligent multimodal retrieval module based on late interaction retrieval, a contrastive multitask classification network for predicting artist, genre, style, media, and tags, and an agentic reasoning strategy enabled through in-context examples for complex visual question answering and artwork explanation via Qwen2.5-VL. Central to this approach is WikiFragments, a Wikipedia-scale dataset of image-text fragments curated to support knowledge-grounded multimodal reasoning. Our framework achieves state-of-the-art results on multiple benchmarks, including a +8.4% F1 improvement in style classification over GraphCLIP and a +7.1 BLEU@1 gain in captioning on ArtPedia. Qualitative analyses show that ArtSeek can interpret visual motifs, infer historical context, and retrieve relevant knowledge, even for obscure works. Though focused on visual arts, our approach generalizes to other domains requiring external knowledge, supporting scalable multimodal AI research. Both the dataset and the source code will be made publicly available at https://github.com/cilabuniba/artseek.

艺术理解多模态知识检索视觉问答

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。