让大模型学会概念层级关系，提升视觉识别一致性。
Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
- 通过文本引导构建分层视觉特征，注入分类知识。
- 在双曲空间中约束层次结构，增强语义相似项的分离度。
- 适合作为插件用于各类多模态模型，提升分类准确性。
分类体系揭示了视觉与语言中概念间的语义关系及内在组织结构。尽管大型多模态模型（LMMs）具备强大能力，却常缺乏分类知识，导致层次化视觉识别（HVR）一致性低。这些模型在微调时仅依赖语言建模目标，缺少显式的分类意识正则化。为此，我们提出层级表示正则化（HiR²），一种简单易用的插件式正则化器，以提升LMMs的层级一致性。具体地，我们设计了一种语义感知的视觉树构建框架，利用文本提示从LLM中间层提取由粗到精的视觉特征。该正则化器包含两个互补目标：基于洛伦兹模型中双曲蕴含锥的分类蕴含损失，强制执行层级结构；以及在单位球面上促进语义相似嵌入的角分离的判别分散损失，同时不破坏径向层级结构。大量实验表明，HiR²能有效捕捉多种LMMs和微调方法下的分类结构。代码已公开于https://github.com/PKU-ICST-MIPL/HiR2_ICML2026。
原文摘要 · Abstract (English)
Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization ($HiR^2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that $HiR^2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2_ICML2026.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。