用语言描述的稀疏编码器，拆解生成图像中的视觉模式
Human-like Content Analysis for Generative AI with Language-Grounded Sparse Encoders
- 将图像分解为可语言描述的视觉模式，实现细粒度分析
- 发现超5000个模式，人类判断一致率达93%
- 适合医疗影像等高风险领域，也适用于蛋白结构分析
生成式AI的快速发展改变了内容创作与人类发展，但在高风险领域引发深远担忧，亟需严格的分析与评估方法。现有方法常将图像视为整体，但真实故障多表现为特定视觉模式，易逃过整体检测，需更细粒度的分解分析。本文提出语言引导的稀疏编码器（LanSE），将图像分解为可解释的视觉模式，并附带自然语言描述。借助可解释模块与大型多模态模型，LanSE可自动识别数据模态中的视觉模式。该方法发现超过5000个视觉模式，人类一致性达93%，在分解评估上优于现有方法，首次系统评估物理合理性，并扩展至医学影像场景。其提取语言关联模式的能力可自然适配生物学、地理学及其他模态（如蛋白质结构、时间序列），推动生成式AI内容分析发展。
原文摘要 · Abstract (English)
The rapid development of generative AI has transformed content creation, communication, and human development. However, this technology raises profound concerns in high-stakes domains, demanding rigorous methods to analyze and evaluate AI-generated content. While existing analytic methods often treat images as indivisible wholes, real-world AI failures generally manifest as specific visual patterns that can evade holistic detection and suit more granular and decomposed analysis. Here we introduce a content analysis tool, Language-Grounded Sparse Encoders (LanSE), which decompose images into interpretable visual patterns with natural language descriptions. Utilizing interpretability modules and large multimodal models, LanSE can automatically identify visual patterns within data modalities. Our method discovers more than 5,000 visual patterns with 93\% human agreement, provides decomposed evaluation outperforming existing methods, establishes the first systematic evaluation of physical plausibility, and extends to medical imaging settings. Our method's capability to extract language-grounded patterns can be naturally adapted to numerous fields, including biology and geography, as well as other data modalities such as protein structures and time series, thereby advancing content analysis for generative AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。