用语言生成动作序列，零样本实现精细视频分类
ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment
- 用大模型生成动作子步骤序列，通过动态时间规整对齐视频帧
- 在ActionAtlas上达30.5%准确率，超越百亿参数视频模型
- 无需训练，适合跨领域细粒度动作识别任务
针对极细粒度动作（如篮球中的Windmill Dunk）的零样本视频分类问题，我们提出ActAlign，一种无需训练、无需视频样本或时间标注的真正零样本方法。该方法将视频分类建模为序列对齐问题，利用大语言模型生成每个类别的有序子动作序列，并在共享嵌入空间中通过动态时间规整（DTW）与视频帧对齐。不依赖任何视频-文本监督或微调，ActAlign在包含多运动场景的最多样化细粒度动作基准ActionAtlas上达到30.5%的准确率，远超人类水平（61.6%），同时仅使用8倍于对比模型的参数量。该方法具备模型无关性和领域通用性，证明了结构化语言先验结合经典对齐方法，可释放图像-语言模型在细粒度视频理解中的开放集识别潜力。
原文摘要 · Abstract (English)
We address the task of zero-shot video classification for extremely fine-grained actions (e.g., Windmill Dunk in basketball), where no video examples or temporal annotations are available for unseen classes. While image-language models (e.g., CLIP, SigLIP) show strong open-set recognition, they lack temporal modeling needed for video understanding. We propose ActAlign, a truly zero-shot, training-free method that formulates video classification as a sequence alignment problem, preserving the generalization strength of pretrained image-language models. For each class, a large language model (LLM) generates an ordered sequence of sub-actions, which we align with video frames using Dynamic Time Warping (DTW) in a shared embedding space. Without any video-text supervision or fine-tuning, ActAlign achieves 30.5% accuracy on ActionAtlas--the most diverse benchmark of fine-grained actions across multiple sports--where human performance is only 61.6%. ActAlign outperforms billion-parameter video-language models while using 8x fewer parameters. Our approach is model-agnostic and domain-general, demonstrating that structured language priors combined with classical alignment methods can unlock the open-set recognition potential of image-language models for fine-grained video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。