分析社交媒体视频中的气候主题，发现视觉聚类可揭示隐含模式。
ClimateVID -- Social Media Videos Analysis and Challenges Involved
- 用零样本分类和聚类方法分析社交媒体视频内容
- DINOv2 和 ConvNeXt V2 能生成有意义的视觉聚类，风格差异更明显
- 适合关注视频内容分析与无监督学习的研究者
社交媒体上短视频的兴起深刻改变了公共讨论的方式。本文通过评估零样本图像分类与聚类能力，推动自动化视觉主题检测。首先，对比 VideoChatGPT、PandaGPT、VideoLLava 与基于帧的 CLIP 基线在零样本分类上的表现；其次，将聚类建模为最小代价多切问题，实现无监督模式发现。尽管当前视觉语言模型尚无法识别特定气候类别，但 DINOv2 与 ConvNeXt V2 生成的聚类结果具有实际意义：前者侧重风格与抽象类别差异，后者体现更细粒度的视觉区分。研究提供详尽评估与实践指导。代码已开源：https://github.com/KathPra/ClimateVID.git。
原文摘要 · Abstract (English)
The pervasive growth of digital content, specifically short videos on social media platforms, has significantly altered how topics are discussed and understood in public discourse. In this work, we advance automated visual theme detection by assessing zero-shot and clustering capabilities on social media data. (1) We evaluated the capabilities of notable VLMs such as VideoChatGPT, PandaGPT, and VideoLLava using zero-shot image classification and compared their performance to the baseline provided by frame-wise CLIP image classification. (2) By treating clustering as a minimum cost multicut problem, we aim to uncover insightful patterns in an unsupervised manner. For both analysis strategies, we provide extensive evaluations and practical guidance to practitioners. While VLMs are currently not able to detect climate change specific classes, the clustering results are distinct visual frames. %Given that VLMs are not currently capable to grasp the climate change discourse, we focus the clustering evaluation of image embedding models. We find that both ConvNeXt V2 and DINOv2 produce meaningful clusters, with DINOv2 focusing more on style differences and abstract categories, while ConvNeXt V2 clusters differ in more fine-grained ways. Code available at https://github.com/KathPra/ClimateVID.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。