用大模型纯文本推理实现人物交互检测，效果翻倍且无需额外模块。
HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection
- 利用大模型的文本推理能力，不依赖检测模块完成交互识别。
- 在HICO-DET上使Qwen2.5-VL-3B准确率提升2倍，泛化性强。
- 适合对轻量化、通用性交互理解有需求的研究者和开发者。
当前的人物交互检测（HOID）方法高度依赖视觉语言模型（VLMs）提供先验知识以增强交互识别能力。然而，如何将VLMs的知识与目标检测器生成的实例表征结合，其训练策略与模型架构仍具挑战性，整体框架复杂，不利于后续开发与应用。另一方面，多模态大语言模型（MLLMs）在人-物交互检测中的内在推理能力尚未充分探索。受近期基于强化学习（RL）训练MLLMs成功的启发，本文提出HOI-R1，首次在无需额外检测模块的情况下，探索语言模型在HOID任务上的潜力。我们设计了交互推理流程与HOID奖励函数，通过纯文本方式解决该任务。在多个开源MLLMs（包括Qwen-VL系列：Qwen2.5-VL和Qwen3-VL，以及Rex-Omni）上进行实验，结果一致提升。尤其，HOI-R1使Qwen2.5-VL-3B的准确率提升2倍，并展现出出色的泛化能力。源代码已公开于https://github.com/cjw2021/HOI-R1。
原文摘要 · Abstract (English)
Recent human-object interaction detection (HOID) methods highly require prior knowledge from vision-language models (VLMs) to enhance the interaction recognition capabilities. The training strategies and model architectures for connecting the knowledge from VLMs to the HOI instance representations from the object detector are challenging, and the whole framework is complex for further development or application. On the other hand, the inherent reasoning abilities of multimodal large language models (MLLMs) on human-object interaction detection are under-explored. Inspired by the recent success of training MLLMs with reinforcement learning (RL) methods, we propose HOI-R1 and first explore the potential of the language model on the HOID task without any additional detection modules. We introduce an HOI reasoning process and HOID reward functions to solve the HOID task by pure text. Experiments on HICO-DET across multiple open-source MLLMs, including the Qwen-VL family (Qwen2.5-VL and Qwen3-VL) and Rex-Omni, show consistent improvements. Especially, HOI-R1 boosts Qwen2.5-VL-3B 2$\times$ accuracy with great generalization ability. The source code is available at https://github.com/cjw2021/HOI-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。