arXiv:2410.02152cs.CV2024-10被引 3

用合成视频训练大模型做手势识别,效果不如真实数据。

An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos

  • 用合成视频生成特征,直接用k近邻分类
  • 合成数据效果差,真实数据少量就够用
  • 微调过的模型更优,自然语言描述难表达手势

本文探索使用合成视频数据与大预训练模型进行基于视频的手势识别的可行性。我们考察这些模型是否具备足够鲁棒和丰富的表征空间,以实现“无需训练”的分类。具体而言,我们采用多种先进的视频编码器从仅由合成视频构成的训练数据中提取特征,用于k近邻分类;并与另一种无需训练的方法——基于手势文本描述的零样本分类进行对比。在RoCoG-v2数据集上的实验表明,仅使用合成训练视频时,在真实测试视频上的分类准确率显著低于使用少量真实训练视频的结果。此外,曾在分类任务上微调过的视频主干网络作为特征提取器表现更优,且微调数据的选择对k近邻性能有显著影响。最后,我们发现基于自然语言描述的零样本分类在手势识别任务上表现不佳,因为手势难以通过语言有效描述。

原文摘要 · Abstract (English)

In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.

手势识别合成数据零样本视频编码器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。