提升大模型对复杂动作的理解能力，提出新框架与训练方法。
LLaVAction: evaluating and training multi-modal large language models for action understanding
- 构建新基准EPIC-KITCHENS-100-MQA，用专家模型生成干扰答案测试模型短板。
- 引入动作标记与两阶段流程，使模型在动作识别上比GPT-4o高21分。
- 适合研究多模态大模型、动作理解及视频分析的学者与开发者。
理解人类行为需精准捕捉动作。由于行为复杂，最适合用语言等语义结构表达。新兴多模态大语言模型（MLLMs）虽具潜力，但其细粒度动作理解能力尚未充分验证。本文将最大的自然视角动作识别数据集EPIC-KITCHENS-100重构为适用于MLLM的基准（EPIC-KITCHENS-100-MQA），并发现当使用专家模型生成困难干扰项时，主流MLLMs难以准确识别动作。为此，我们构建了包含‘难例’动作识别、时间定位、描述生成和自由问答的监督微调数据集，以增强模型多样性理解能力。提出新模型LLaVAction，通过引入动作标记增强对视觉标记的关注，并采用两阶段流水线提取结构化动作。实验表明，该方法显著提升模型性能，在MLLM基准上较GPT-4o提高21个百分点，在传统动作识别基准上也有显著改进，证明其为复杂动作任务的有效路径。代码、数据、基准与模型已开源。
原文摘要 · Abstract (English)
Understanding human behavior requires measuring behavioral actions. Due to its complexity, behavior is best mapped onto a rich, semantic structure such as language. Emerging multimodal large language models (MLLMs) are promising candidates, but their fine-grained action understanding ability has not been fully examined. In this work, we reformulate EPIC-KITCHENS-100, one of the largest and most challenging egocentric action recognition datasets, into a MLLM benchmark (EPIC-KITCHENS-100-MQA). We demonstrate that when we sample difficult answers based on specialist models as distractors, leading MLLMs struggle to recognize the correct actions. How can we increase the performance of MLLMs? We curated a supervised finetuning dataset that includes `hard' action recognition, temporal detection, captioning, and free-form question answering to improve models' diverse action understanding capabilities. We introduce a new model called LLaVAction that adds an action token to boost models' attention on visual tokens and a two-stage pipeline to obtain structured actions. LLaVAction greatly improves the MLLMs' ability of action understanding, achieving strong improvements on both MLLM benchmarks (21 points in accuracy over GPT-4o on EPIC-KITCHENS-100-MQA) and established action recognition benchmarks, suggesting that our methods prepare MLLMs to be a promising path forward for complex action tasks. Code, data, the benchmark, and models are available at https://github.com/AdaptiveMotorControlLab/LLaVAction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。