用开放文章和维基知识为艺术作品自动添加背景信息。
ArtContext: Contextualizing Artworks with Open-Access Art History Articles and Wikidata Knowledge through a LoRA-Tuned CLIP Model
- 用LoRA微调CLIP模型，适配艺术史领域。
- 新模型PaintingCLIP在艺术上下文理解上优于原始CLIP。
- 可扩展至其他人文学科，无需大量标注数据。
许多艺术史文章讨论艺术品的整体特征及具体部分，如构图、图像学或物质文化。然而，在查看一件艺术品时，难以快速识别相关文章中提及的内容。为此，我们提出ArtContext，一个将开放获取的艺术史文章与维基数据知识结合，为艺术品打标相关信息的流程。通过新型语料收集管道，我们使用低秩适应（LoRA）微调CLIP模型，构建出专用于艺术领域的模型PaintingCLIP。该模型在弱监督语料上训练，显著优于原始CLIP，在提供艺术品上下文方面表现更佳。所提流程具有通用性，可轻松应用于众多人文学科。
原文摘要 · Abstract (English)
Many Art History articles discuss artworks in general as well as specific parts of works, such as layout, iconography, or material culture. However, when viewing an artwork, it is not trivial to identify what different articles have said about the piece. Therefore, we propose ArtContext, a pipeline for taking a corpus of Open-Access Art History articles and Wikidata Knowledge and annotating Artworks with this information. We do this using a novel corpus collection pipeline, then learn a bespoke CLIP model adapted using Low-Rank Adaptation (LoRA) to make it domain-specific. We show that the new model, PaintingCLIP, which is weakly supervised by the collected corpus, outperforms CLIP and provides context for a given artwork. The proposed pipeline is generalisable and can be readily applied to numerous humanities areas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。