用大模型理解力无监督训练小模型，实现精准人物交互检测
CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models
- 从大模型图像级理解中蒸馏实例级交互信息，无需人工标注
- 在HICO-DET和V-COCO上超越弱监督与有监督方法
- 适合缺乏标注数据的交互检测场景，尤其适用于轻量级部署
人体-物体交互（HOI）检测近年来借助视觉语言模型（VLM）取得进展，但这些方法通常依赖大量人工标注。视觉大语言模型（VLLM）虽能内在识别和推理图像级交互，却计算开销大且不专用于实例级HOI检测。为此，我们提出跨层级HOI蒸馏（CL-HOI）框架，无需人工标注即可从VLLM的图像级理解中蒸馏出实例级交互信息。该方法包含两个阶段：上下文蒸馏阶段，通过视觉语言转换器（VLT）将视觉信息转化为语言形式；交互蒸馏阶段，通过交互认知网络（ICN）推理空间、视觉与上下文关系。设计对比蒸馏损失函数，将教师模型的图像级上下文与交互知识传递至学生模型，实现实例级HOI检测。在HICO-DET和V-COCO数据集上的评估表明，我们的CL-HOI超越现有弱监督方法及基于VLLM的监督方法，验证了其在无手动标签条件下检测HOI的有效性。
原文摘要 · Abstract (English)
Human-object interaction (HOI) detection has seen advancements with Vision Language Models (VLMs), but these methods often depend on extensive manual annotations. Vision Large Language Models (VLLMs) can inherently recognize and reason about interactions at the image level but are computationally heavy and not designed for instance-level HOI detection. To overcome these limitations, we propose a Cross-Level HOI distillation (CL-HOI) framework, which distills instance-level HOIs from VLLMs image-level understanding without the need for manual annotations. Our approach involves two stages: context distillation, where a Visual Linguistic Translator (VLT) converts visual information into linguistic form, and interaction distillation, where an Interaction Cognition Network (ICN) reasons about spatial, visual, and context relations. We design contrastive distillation losses to transfer image-level context and interaction knowledge from the teacher to the student model, enabling instance-level HOI detection. Evaluations on HICO-DET and V-COCO datasets demonstrate that our CL-HOI surpasses existing weakly supervised methods and VLLM supervised methods, showing its efficacy in detecting HOIs without manual labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。