用测试时校准提升大模型对微手势的识别能力
Zero-MELO: Test-Time Evidence Calibration with Multimodal LLMs for Zero-Shot Micro-Gesture Recognition

- 通过树搜索逐步获取局部视觉证据,增强细粒度感知
- 在iMiGUE和MA-52数据集上准确率分别达26.84%和22.10%
- 适合做零样本微手势识别与情感分析的研究者参考
虽然多模态大语言模型(MLLMs)在通用视频理解中表现优异,但在细粒度、以运动为中心的任务中仍存在局限。这一问题在微手势识别(MGR)中尤为突出:微手势(MGs)是短暂、局部且微妙的人体动作,常用于隐性情感分析,但易被常规提示忽略。尽管已有大量判别方法研究MGR,MLLM在该任务上的应用仍不充分,性能较差。我们假设MLLM的运动敏感表征受限于单次前向推理机制,可通过精心设计的测试时引导显著提升。基于此前关于视频LLM时间不敏感性的发现,我们在负对数似然(NLL)空间诊断零样本MGR错误,发现两大瓶颈:1)局部证据不足;2)由语言和运动无关外观引发严重得分偏差。为此,我们提出一种新型测试时证据校准框架,结合树搜索机制逐步获取细粒度视觉证据,并引入校准模块缓解得分偏差。多线索融合模块整合多种线索,避免依赖单一线索做出最终预测。该框架在iMiGUE和MA-52数据集上达到26.84%和22.10%的平均类别准确率,显著优于Qwen2.5-VL基线(分别为16.15%和10.20%)。代码将发布于https://zero-melo.github.io/Zero-MELO。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) excel in general video understanding, their capability in fine-grained and motion-centric tasks remains limited. This limitation is particularly critical in micro-gesture recognition (MGR), where micro-gestures (MGs) - subtle, short-duration, and spatially localized human movements - serve as key discriminative signals for implicit affective analysis, yet are easily neglected following common prompting practices. Although MGR has been intensively studied by many discriminative approaches, the use of MLLMs for MGR is underexplored, with notably poor performance. We hypothesize that the motion-sensitive representation ability of MLLMs is constrained by their inherent single-pass forward inference, which can be substantially enhanced through carefully designed test-time guidance. Motivated by this, building on our prior findings regarding temporal insensitivity in Video LLMs, we diagnose zero-shot MGR errors in the Negative Log-Likelihood (NLL) space. We observe that MLLMs suffer from two bottlenecks: 1) insufficient localized evidence and 2) severe score biases driven by language and motion-agnostic appearances. Thus, we propose a novel test-time evidence calibration framework that improves both reasoning details and prediction reliability. Specifically, we introduce a tree search mechanism to progressively acquire localized, fine-grained visual evidence, coupled with a test-time calibration module to mitigate score biases. The multi-cue fusion module then integrates evidence from multiple cues without relying on a single cue for final prediction. Our framework achieves mean-class accuracies of 26.84\% on iMiGUE and 22.10\% on MA-52, significantly outperforming the Qwen2.5-VL baseline, which produces 16.15\% and 10.20\%, respectively. The code will be available at https://zero-melo.github.io/Zero-MELO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。