统一处理人中心指代任务,让AI理解对话中的人是谁
RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarios
- 用序列合并将图像文本等数据转为统一语义标记
- 在多个任务上表现优异,支持复杂推理
- 适合聊天机器人、体育分析等需要理解人的场景
人中心感知在真实应用中至关重要。尽管近期研究取得显著进展,但大多局限于视觉领域,缺乏与人类指令的交互,限制了在聊天机器人和体育分析等场景的应用。本文提出指代性人中心感知任务,通过指代提示定位图像中的特定人物。为此,我们提出RefHCM(指代人中心模型),一个统一框架,整合多种人中心指代任务。RefHCM采用序列合并技术,将图像、文本、坐标、分割图等多模态原始数据转换为语义标记,形成标准化表示。该表示使多样化的任务可统一建模为序列到序列问题,使用标准编码器-解码器Transformer架构求解。得益于统一学习策略,RefHCM有效促进任务间知识迁移,并展现出处理复杂推理的意外能力。本工作首次以通用框架解决指代性人中心感知问题,同时建立了对应基准,设定新标准。大量实验表明,RefHCM在多个任务上表现竞争力甚至更优。代码与数据已公开于https://github.com/JJJYmmm/RefHCM。
原文摘要 · Abstract (English)
Human-centric perceptions play a crucial role in real-world applications. While recent human-centric works have achieved impressive progress, these efforts are often constrained to the visual domain and lack interaction with human instructions, limiting their applicability in broader scenarios such as chatbots and sports analysis. This paper introduces Referring Human Perceptions, where a referring prompt specifies the person of interest in an image. To tackle the new task, we propose RefHCM (Referring Human-Centric Model), a unified framework to integrate a wide range of human-centric referring tasks. Specifically, RefHCM employs sequence mergers to convert raw multimodal data -- including images, text, coordinates, and parsing maps -- into semantic tokens. This standardized representation enables RefHCM to reformulate diverse human-centric referring tasks into a sequence-to-sequence paradigm, solved using a plain encoder-decoder transformer architecture. Benefiting from a unified learning strategy, RefHCM effectively facilitates knowledge transfer across tasks and exhibits unforeseen capabilities in handling complex reasoning. This work represents the first attempt to address referring human perceptions with a general-purpose framework, while simultaneously establishing a corresponding benchmark that sets new standards for the field. Extensive experiments showcase RefHCM's competitive and even superior performance across multiple human-centric referring tasks. The code and data are publicly at https://github.com/JJJYmmm/RefHCM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。