用自动生成的描述代替预设概念,让模型解释更自然且无信息泄漏。
Caption Bottleneck Models

- 用大模型生成的图像描述替代固定概念层,实现自由语言表达
- 在多个数据集上达到与传统方法相当的准确率,同时保证解释可读性
- 自动发现适合特定数据集的高质量概念,无需人工标注或外部词典
概念瓶颈模型(CBM)通过将预测过程经由人类可理解的概念层来提升可解释性。然而,为特定数据集定义最优概念集仍是未解难题。现有方法依赖昂贵的人工标注或仅基于类别名的LLM生成列表。即使“开放词汇”变体也通常依赖静态概念集,限制了新概念发现并引入标签偏差。此外,传统CBM常存在信息泄漏问题,未建模的视觉特征绕过瓶颈,破坏解释完整性。为此,我们提出标题瓶颈模型(CaBM),通过用自由形式的自然语言取代固定概念层,以LMM生成的图像描述表示图像,并严格基于该文本训练分类器,从结构上杜绝信息泄漏。同时,通过训练后分析文本分类器,CaBM能自主发现高质量、数据集相关的概念。在细粒度和粗粒度基准测试中,CaBM实现了具有竞争力的准确率,同时保持可解释性,且无需依赖外部词典或人工标注。
原文摘要 · Abstract (English)
Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。