arXiv:2501.13341cs.CV2025-01CVPR被引 4

用大模型提取多维度知识,提升图像分类性能

Multi-aspect Knowledge Distillation with Large Language Model

  • 通过多角度提问大模型获取多维知识
  • 扩展输出维度并融合交叉熵与二元交叉熵损失
  • 适合希望提升模型泛化能力的研究者

深度学习的进步显著提升了计算机视觉任务的性能。传统图像分类方法主要通过修改模型结构或添加特征,并使用类别logits上的交叉熵损失进行优化,但这类方法仅关注类别标签,难以学习类别的多种特征(如自然位置、形状变化等)。为此,本文提出一种基于多模态大语言模型(MLLM)的多方面知识蒸馏方法:1)向大模型提出与目标知识相关的多维度问题;2)提取MLLM生成的对应logits;3)扩展学生模型输出维度以蒸馏这些多维度logits。训练时同时使用交叉熵损失(针对类别)和二元交叉熵损失(针对多维度)。实验表明,该方法在图像分类任务中优于基线模型,且能有效传递多种视觉及抽象层面的知识,显著提升模型性能。此外,方法具备扩展至目标检测等任务的潜力。

原文摘要 · Abstract (English)

Recent advancements in deep learning have significantly improved performance on computer vision tasks. Previous image classification methods primarily modify model architectures or add features, and they optimize models using cross-entropy loss on class logits. Since they focus on classifying images with considering class labels, these methods may struggle to learn various \emph{aspects} of classes (e.g., natural positions and shape changes). Rethinking the previous approach from a novel view, we propose a multi-aspect knowledge distillation method using Multimodal Large Language Models (MLLMs). Our approach involves: 1) querying Large Language Model with multi-aspect questions relevant to the knowledge we want to transfer to the model, 2) extracting corresponding logits from MLLM, and 3) expanding the model's output dimensions to distill these multi-aspect logits. We then apply cross-entropy loss to class logits and binary cross-entropy loss to multi-aspect logits. Through our method, the model can learn not only the knowledge about visual aspects but also the abstract and complex aspects that require a deeper understanding. We primarily apply our method to image classification, and to explore the potential for extending our model, such as object detection. In all experimental results, our method improves the performance of the baselines. Additionally, we analyze the effect of multi-aspect knowledge distillation. These results demonstrate that our method can transfer knowledge about various aspects to the model and the aspect knowledge can enhance model performance in computer vision tasks.

知识蒸馏大模型图像分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。