让大模型理解生物分类层级,提升对新物种的识别能力
Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models
- 用生物基础模型的层级表征对齐视觉特征,注入分类知识
- 在复杂生物分类体系中,新旧类别识别准确率均显著提升
- 适合需要跨层级理解、识别未知类别的科研与生物信息场景
高性能通用视觉理解模型应能将视觉输入映射到标签的分类树,并识别训练集外、图像稀缺的新类别。大模型在已知类别的细粒度识别上表现优异,但在从粗到细的层级识别任务中仍受限,尤其面对新类别时表现不足。为此,我们提出税理结构感知表示对齐(TARA),一种简单有效的策略,通过生物基础模型(BFMs)的层次对比学习所提取的丰富生物关系表征,对齐视觉特征中间层表示,促使大模型提取出结构化于分类树中的判别性视觉线索。同时,将首个回答标记的表征与真实标签对齐,灵活衔接上下文视觉特征与不同粒度的类别,适配用户意图。实验表明,TARA持续提升大模型在层级一致性与叶节点准确率上的表现,实现对复杂生物分类体系中已知与新类别可靠识别。代码开源:https://github.com/PKU-ICST-MIPL/TARA_CVPR2026。
原文摘要 · Abstract (English)
A high-performing, general-purpose visual understanding model should map visual inputs to a taxonomic tree of labels, identify novel categories beyond the training set for which few or no publicly available images exist. Large Multimodal Models (LMMs) have achieved remarkable progress in fine-grained visual recognition (FGVR) for known categories. However, they remain limited in hierarchical visual recognition (HVR) that aims at predicting consistent label paths from coarse to fine categories, especially for novel categories. To tackle these challenges, we propose Taxonomy-Aware Representation Alignment (TARA), a simple yet effective strategy to inject taxonomic knowledge into LMMs. TARA leverages representations from biology foundation models (BFMs) that encode rich biological relationships through hierarchical contrastive learning. By aligning the intermediate representations of visual features with those of BFMs, LMMs are encouraged to extract discriminative visual cues well structured in the taxonomy tree. Additionally, we align the representations of the first answer token with the ground-truth label, flexibly bridging the gap between contextualized visual features and categories of varying granularity according to user intent. Experiments demonstrate that TARA consistently enhances LMMs' hierarchical consistency and leaf node accuracy, enabling reliable recognition of both known and novel categories within complex biological taxonomies. Code is available at https://github.com/PKU-ICST-MIPL/TARA_CVPR2026.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。