arXiv:2505.05681cs.CV2025-05被引 3

用无标签视频训练猿类行为检索模型,提升搜索准确率超100%。

Fine-Tuning Video-Text Contrastive Model for Primate Behavior Retrieval from Unlabeled Raw Videos

  • 自动提取视频与弱音频描述的对齐片段,构建高质量训练数据
  • 在自建数据集上,8帧和16帧模型的Hits@5提升超114%和167%
  • 适合灵长类行为研究者快速检索野外视频中的特定行为

在自然栖息地拍摄的非人类灵长类动物视频是研究其野外行为的重要数据源。本文针对卷尾猴这一特定领域,微调预训练的视频-文本基础模型,旨在帮助研究人员从海量原始视频中高效检索有用片段。面对仅依赖未标注视频及偶尔提供的弱音频描述的挑战,我们利用多模态大语言模型(MLLM)和视觉-语言模型(VLM)的优势,提出双重策略:一个智能数据处理流水线与微调流程。该流水线能自动从原始视频中提取出语义对齐的干净视频-文本对,随后使用低秩适应(LoRA)微调微软的X-CLIP模型。在自建数据集上,8帧模型的Hits@5提升114%,16帧模型提升167%。基于NDCG@K指标,本模型能有效排序多数行为,而原始预训练模型则无法做到。代码将在论文录用后公开。

原文摘要 · Abstract (English)

Video recordings of nonhuman primates in their natural habitat are a common source for studying their behavior in the wild. We fine-tune pre-trained video-text foundational models for the specific domain of capuchin monkeys, with the goal of developing useful computational models to help researchers to retrieve useful clips from videos. We focus on the challenging problem of training a model based solely on raw, unlabeled video footage, using weak audio descriptions sometimes provided by field collaborators. We leverage recent advances in Multimodal Large Language Models (MLLMs) and Vision-Language Models (VLMs) to address the extremely noisy nature of both video and audio content. Specifically, we propose a two-folded approach: an agentic data treatment pipeline and a fine-tuning process. The data processing pipeline automatically extracts clean and semantically aligned video-text pairs from the raw videos, which are subsequently used to fine-tune a pre-trained Microsoft's X-CLIP model through Low-Rank Adaptation (LoRA). We obtained an uplift in $Hits@5$ of $167\%$ for the 16 frames model and an uplift of $114\%$ for the 8 frame model on our domain data. Moreover, based on $NDCG@K$ results, our model is able to rank well most of the considered behaviors, while the tested raw pre-trained models are not able to rank them at all. The code will be made available upon acceptance.

行为检索视频理解弱监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。