让轻量视觉语言模型自问自答,精准理解人类细微动作。
Gold Points Sniper: Self-guided Visual Reasoning in VLM for Fine-grained Action Understanding

- 通过自引导提问机制,聚焦动作关键细节。
- 在CAP数据集上性能接近GPT-4o,且事实准确率更高。
- 适合家庭机器人需精细理解人类行为的场景。
在日常环境中运行的机器人需从宽视角中理解人类的细粒度动作、意图和上下文线索,而当前系统尚无法满足此需求。尽管开放词汇动作识别方法仍局限于预定义标签,视觉语言模型(VLMs)在输出的信息丰富性与事实准确性之间存在固有权衡,均难以实现可靠的语义解析。本文提出黄金点狙击者(Gold Points Sniper, GPS),一种新框架,赋予轻量级VLM自引导多模态推理能力,以实现细粒度人类动作理解。该框架包含三个核心模块:黄金点提取器训练VLM识别关键动作相关细节;选择性苏格拉底提问器通过选择性自问,验证并精炼这些细节;语义蕴含评估器利用语义蕴含分类定量评估事实一致性。在基于CAP基准构建的指令微调数据集上的大量实验表明,经GPS增强的轻量级VLM性能显著提升,部分模型表现接近专有的GPT-4o,同时保持更优的事实准确性。本工作为家用机器人中的细粒度动作理解奠定了可靠基础,使机器人能通过信息密集且事实可靠的描述安全解读人类行为。源代码、训练配置、标注提示及数据集详情已公开于https://github.com/Haodi-Liu/GPS-Gold-Point-Sniper。
原文摘要 · Abstract (English)
Robots operating in everyday environments must understand fine-grained human actions, intentions, and contextual cues from broad views where people occupy only small regions, a capability unmet by current systems. While open-vocabulary action recognition methods remain limited to assigning predefined labels, and vision-language models (VLMs) face an inherent trade-off between informational richness and factual fidelity in their outputs, neither approach achieves the deep semantic interpretation required for reliable human-robot interaction. We propose Gold Points Sniper (GPS), a novel framework that empowers lightweight VLMs with self-guided multimodal reasoning capabilities for fine-grained human action understanding. Our approach comprises three key modules: Gold Points Extractor trains VLMs to identify critical action-relevant details, Selective Socratic Questioner validates and refines these details through selective self-questioning, and Semantic Entailment Evaluator quantitatively assesses factual consistency using semantic entailment classification. Extensive experiments on our curated instruction-tuning dataset based on the CAP benchmark demonstrate that GPS-enhanced lightweight VLMs achieve substantial performance improvements, with some models reaching performance comparable to proprietary GPT-4o while maintaining superior factual accuracy. Our work establishes a reliable foundation for fine-grained action understanding in domestic robotics, enabling robots to safely interpret human behavior through information-dense yet factually grounded descriptions. Source code, training configurations, annotation prompts, and dataset details are released at https://github.com/Haodi-Liu/GPS-Gold-Point-Sniper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。