构建城市环境理解新基准,评估大模型对城市变迁与主观感知的把握能力。
UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective
- 设计三维度14.3K问题,涵盖静态感知、时间变化与主观评价
- 20个顶尖多模态模型测试,Gemini-2.5 Pro接近人类专家水平(差值仅1.5%)
- 模型在细节变化检测上超人,在长期时空推理上仍显不足
城市化影响全球超半数人口,以人为中心理解其结构与感知变化对可持续发展至关重要。尽管多模态大语言模型(MLLMs)在多个领域表现卓越,现有针对城市环境的评测基准仍有限,缺乏对时间演化和符合人类感知的主观体验的系统性考察。为此,我们提出UrbanFeel,一个全面评估MLLM在城市演进理解与主观环境感知方面表现的基准。该基准包含14.3K个精心构建的视觉问题,覆盖三个认知递进维度:静态场景感知、时间变化理解与主观环境感知。数据源自11个代表性城市的多时相单视角与全景街景图像,通过空间聚类、规则生成、模型辅助提示与人工标注相结合的混合流程生成高质量问答对。对20个先进MLLM的广泛评估显示,Gemini-2.5 Pro综合表现最佳,准确率逼近人类专家水平,平均差距仅为1.5%。多数模型在基于场景理解的任务中表现良好,部分模型甚至在像素级变化检测上超越人工标注者。然而,在需长期时间推理的城市发展任务中性能明显下降。此外,在主观感知维度,若干模型在美观度与安全性等评价上达到或超过人类一致性水平。
原文摘要 · Abstract (English)
Urban development impacts over half of the global population, making human-centered understanding of its structural and perceptual changes essential for sustainable development. While Multimodal Large Language Models (MLLMs) have shown remarkable capabilities across various domains, existing benchmarks that explore their performance in urban environments remain limited, lacking systematic exploration of temporal evolution and subjective perception of urban environment that aligns with human perception. To address these limitations, we propose UrbanFeel, a comprehensive benchmark designed to evaluate the performance of MLLMs in urban development understanding and subjective environmental perception. UrbanFeel comprises 14.3K carefully constructed visual questions spanning three cognitively progressive dimensions: Static Scene Perception, Temporal Change Understanding, and Subjective Environmental Perception. We collect multi-temporal single-view and panoramic street-view images from 11 representative cities worldwide, and generate high-quality question-answer pairs through a hybrid pipeline of spatial clustering, rule-based generation, model-assisted prompting, and manual annotation. Through extensive evaluation of 20 state-of-the-art MLLMs, we observe that Gemini-2.5 Pro achieves the best overall performance, with its accuracy approaching human expert levels and narrowing the average gap to just 1.5\%. Most models perform well on tasks grounded in scene understanding. In particular, some models even surpass human annotators in pixel-level change detection. However, performance drops notably in tasks requiring temporal reasoning over urban development. Additionally, in the subjective perception dimension, several models reach human-level or even higher consistency in evaluating dimension such as beautiful and safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。