用引导提示学习让大模型高效识别未见过的人物交互,参数量仅需对手方法的三分之一。
EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection
- 用LLM和VLM双引导设计可学习提示,融合语义与视觉信息。
- 通过已知类推未知类,使提示学习更适配未见交互类别。
- 参数量减少至33.95%,零样本检测性能领先现有方法。
在零样本设定下检测人物交互(HOI)面临巨大挑战,因模型需处理未见类别。现有方法依赖对齐视觉编码器与大型视觉-语言模型(VLM)以利用其丰富知识,但需庞大计算资源且训练困难。提示学习为替代方案,但在特定数据集微调常导致过拟合已见类别,影响未见类别表现。为此,本文提出高效零样本HOI检测框架EZ-HOI。首先,引入大语言模型(LLM)与VLM双重引导,生成融合详细交互描述与视觉语义的可学习提示,适配VLM至HOI任务。其次,因训练集仅含已见类别标签,微调易优化于已见类,故设计基于相关已见类信息的未见类提示学习策略,利用LLM突出未见类与相关已见类差异。在基准数据集上的定量评估显示,本方法在多种零样本设置下均达当前最优,可训练参数仅占现有方法的10.35%至33.95%。代码已开源。
原文摘要 · Abstract (English)
Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers an alternative to direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due to the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Shot HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics to adapt VLMs to HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations on benchmark datasets demonstrate that our EZ-HOI achieves state-of-the-art performance across various zero-shot settings with only 10.35% to 33.95% of the trainable parameters compared to existing methods. Code is available at https://github.com/ChelsieLei/EZ-HOI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。