arXiv:2606.15417cs.CV2026-06

将视频转为时序图,让视觉语言模型更精准推理第一视角动作。

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

论文配图:From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
图 1 · 摘自论文原文
  • 把视频切片生成自然语言描述,再构造成开放词汇的时序图。
  • 少样本图示例使准确率显著优于直接看帧或纯图推理。
  • 适合想用大模型做动作识别又不想微调的研究者。

第一视角视频中的动作推理需要捕捉手物交互的细微变化,通用视觉语言模型(VLMs)直接处理原始像素时常表现不佳。本文提出将视觉感知与符号推理解耦:将视频转换为时序动作图。通过多阶段提示流程,先在短时窗内生成密集自然语言叙事作为语义瓶颈,再将其形式化为结构化、开放词汇的图表示。在EGTEA和Epic-Kitchens-100数据集上,该符号表示实现了高效的上下文学习:少量图示例即带来显著准确率提升,优于零样本帧和图基推理。即使在零样本设置下,图基推理仍可媲美像素基推理,尽管后者存在预训练偏差优势。对11个来自6个模型家族(2B至235B参数)的开源VLM进行测试,结果表明当前VLM更适合作为符号推理器而非直接视觉观察者。通过将视频投影至语言域,我们提供了一种无需微调、可扩展的替代方案,更好发挥模型潜藏的推理能力。代码将公开。

原文摘要 · Abstract (English)

Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame and graph-based inference alike. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.

动作识别视觉语言模型时序图少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。