arXiv:2604.01586cs.CVcs.AI2026-04中稿 · CVPR被引 1

提出新评估框架SHOE,让模型预测更贴近人类对交互的理解。

SHOE: Semantic HOI Open-Vocabulary Evaluation Metric

  • 将交互预测拆解为动词与对象,用多大模型平均打分计算语义相似度
  • 在HICO-DET上与人工评分一致率达85.73%,优于现有方法
  • 适合评估开放词汇下的视觉-语言交互模型,推动真实场景应用

开放词汇人-物交互检测是构建可扩展系统的关键,能泛化到现实场景中未见的交互,并支持基于语义的多模态推理。然而,传统评估指标(如mAP)将交互类别视为离散标签,无法认可语义合理但表达不同的预测(如“靠在沙发上”与“坐在沙发上”),限制了其在开放词汇预测中的适用性。本文提出SHOE(语义开放词汇评估),通过分解每个交互预测的动词与对象成分,利用多个大语言模型的平均值估算其语义相似度,综合生成匹配分数,实现超越精确字符串匹配的评估。该方法在标准基准数据集HICO-DET上验证,其评分与人类判断高度一致,达成85.73%的平均一致率,显著优于基于LLM和嵌入的基线方法。本工作强调了语义驱动的交互评估对模拟人类理解的重要性,相关评估工具将公开发布以促进后续研究。

原文摘要 · Abstract (English)

Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human-object relationships. However, standard evaluation metrics, such as mean Average Precision (mAP), treat HOI classes as discrete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., "lean on couch" vs. "sit on couch"), limiting their applicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. We introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs), and combines them into a similarity score to evaluate alignment beyond exact string match. This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with the average human ratings. Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions. We will release our evaluation metric to the public to facilitate future research.

HOI检测语义评估大模型开放词汇

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。