用多模态大模型分析数千万张城市图像，自动发现随时间变化的常见趋势。
Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images
- 将海量图像分析拆解为可处理的子任务，结合多模态大模型逐个解决
- 在真实城市图像中发现如‘增设户外餐饮区’等具体变化趋势
- 无需预设标签或目标，适合探索性视觉分析，适合城市研究者使用
我们提出一个系统，利用多模态大模型（MLLMs）分析包含数千万张不同时期图像的大规模数据库，旨在发现城市中频繁出现的时序变化模式。与以往视觉分析不同，本方法能回答开放性问题（如“城市中常见的变化类型有哪些？”），无需预定义目标或训练标签。传统基于学习或无监督的方法难以胜任此类任务。我们发现MLLM具备开放语义理解能力，是理想的分析工具。然而，数据规模比单次输入容量大四数量级，因此我们设计了一种自下而上的分解策略，将复杂问题拆解为可管理的子问题，并为每个子问题构建基于MLLM的解决方案。实验与消融研究显示，该系统显著优于基线方法，成功从大城市图像中识别出如“增设户外餐饮区”“立交桥被涂成蓝色”等有趣趋势。更多结果与交互演示见 https://boyangdeng.com/visual-chronicles。
原文摘要 · Abstract (English)
We present a system using Multimodal LLMs (MLLMs) to analyze a large database with tens of millions of images captured at different times, with the aim of discovering patterns in temporal changes. Specifically, we aim to capture frequent co-occurring changes ("trends") across a city over a certain period. Unlike previous visual analyses, our analysis answers open-ended queries (e.g., "what are the frequent types of changes in the city?") without any predetermined target subjects or training labels. These properties cast prior learning-based or unsupervised visual analysis tools unsuitable. We identify MLLMs as a novel tool for their open-ended semantic understanding capabilities. Yet, our datasets are four orders of magnitude too large for an MLLM to ingest as context. So we introduce a bottom-up procedure that decomposes the massive visual analysis problem into more tractable sub-problems. We carefully design MLLM-based solutions to each sub-problem. During experiments and ablation studies with our system, we find it significantly outperforms baselines and is able to discover interesting trends from images captured in large cities (e.g., "addition of outdoor dining,", "overpass was painted blue," etc.). See more results and interactive demos at https://boyangdeng.com/visual-chronicles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。