构建细粒度动作理解基准,解决视频描述中的幻觉问题
KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understanding
- 用运动学解析+语言分析自动标注动作细节
- 提出新数据集,包含肢体级动作描述与反幻觉评测题
- 设计新算法,可独立评估并减少描述幻觉
尽管近期取得进展,视频字幕模型在准确描述细粒度运动细节方面仍存在显著局限,且普遍存在严重幻觉问题。这一挑战在以动作为核心的视频中尤为突出,精确刻画复杂动作与肢体动态至关重要却常被忽视。为此,我们提出一种自动化标注流程,融合基于运动学的动作计算与语言解析,实现对复杂人体动作的精细分解与描述。基于此流程,我们构建并发布了新的开源数据集——运动学解析动作基准(KPM-Bench),包含:(i) 细粒度视频-字幕对,全面呈现复杂动作中的肢体级动态;(ii) 聚焦动作理解的多样且具挑战性的问答对;(iii) 专为评估动作描述幻觉现象而精心设计的评测集。为进一步系统性缓解幻觉问题,我们提出语言基础的运动解析与提取(MoPE)算法,可直接从文本字幕中精准提取运动属性。借助MoPE,我们建立了一种不依赖大规模视觉-语言或纯语言模型的精确幻觉评估指标。将MoPE集成至GRPO后训练框架,有效缓解了幻觉问题,显著提升了动作核心视频字幕模型的可靠性。
原文摘要 · Abstract (English)
Despite recent advancements, video captioning models still face significant limitations in accurately describing fine-grained motion details and suffer from severe hallucination issues. These challenges become particularly prominent when generating captions for motion-centric videos, where precise depiction of intricate movements and limb dynamics is crucial yet often neglected. To alleviate this gap, we introduce an automated annotation pipeline that integrates kinematic-based motion computation with linguistic parsing, enabling detailed decomposition and description of complex human motions. Based on this pipeline, we construct and release the Kinematic Parsing Motion Benchmark (KPM-Bench), a novel open-source dataset designed to facilitate fine-grained motion understanding. KPM-Bench consists of (i) fine-grained video-caption pairs that comprehensively illustrate limb-level dynamics in complex actions, (ii) diverse and challenging question-answer pairs focusing specifically on motion understanding, and (iii) a meticulously curated evaluation set specifically designed to assess hallucination phenomena associated with motion descriptions. Furthermore, to address hallucination issues systematically, we propose the linguistically grounded Motion Parsing and Extraction (MoPE) algorithm, capable of accurately extracting motion-specific attributes directly from textual captions. Leveraging MoPE, we introduce a precise hallucination evaluation metric that functions independently of large-scale vision-language or language-only models. By integrating MoPE into the GRPO post-training framework, we effectively mitigate hallucination problems, significantly improving the reliability of motion-centric video captioning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。