让视觉语言模型学会层级分类，高效且预测更一致。
Hierarchy-Aware Fine-Tuning of Vision-Language Models
- 用树形路径和兄弟类一致性损失，约束模型预测层级关系。
- 在多个数据集上提升完整路径准确率，降低层级不一致错误。
- 仅微调少量参数，适合资源有限的场景应用。
视觉语言模型（VLMs）通过大规模图文预训练获得强大的多模态表征，但将其适配到层级分类任务仍研究不足。传统方法将标签视为平铺类别，需全量微调，成本高且不同层级预测不一致。本文提出一种高效的层次感知微调框架，仅更新少量参数，同时保证结构一致性。结合两种目标：树路径KL散度（TP-KL）确保沿真实标签路径的预测垂直一致；层次兄弟平滑交叉熵（HiSCE）促进同级类别间预测一致。两类损失均在VLM共享嵌入空间中运行，与轻量级LoRA适配集成。在多个基准测试中，该方法显著提升全路径准确率，降低基于树的不一致误差，参数开销极小。本方法为适配VLM至结构化分类体系提供了高效策略。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) learn powerful multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。