用领域特定提示增强视频问答模型推理能力
Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
- 引入领域特定的实体-动作启发式提示，引导模型关注关键线索
- 在多个视频问答数据集上显著优于现有模型
- 适合需要精准上下文理解的视频分析任务
视频问答（VideoQA）是视频理解与语言处理的关键交叉领域，要求兼具单模态判别能力与复杂的跨模态交互以实现准确推理。尽管多模态预训练模型和视频语言基础模型取得进展，但它们在领域特定视频问答任务中仍表现不佳，原因在于其通用预训练目标无法满足具体推理需求。为此，我们提出HeurVidQA框架，利用领域特定的实体-动作启发式提示来优化预训练视频语言基础模型。该方法将模型视为隐式知识引擎，通过领域特定的实体-动作提示器引导模型聚焦于精确线索，提升对关键实体和动作的识别与解释能力，从而增强推理性能。在多个视频问答数据集上的广泛评估表明，该方法显著优于现有模型，证明了将领域特定知识融入视频语言模型对实现更准确、更具上下文感知能力的视频问答的重要性。
原文摘要 · Abstract (English)
Video Question Answering (VideoQA) represents a crucial intersection between video understanding and language processing, requiring both discriminative unimodal comprehension and sophisticated cross-modal interaction for accurate inference. Despite advancements in multi-modal pre-trained models and video-language foundation models, these systems often struggle with domain-specific VideoQA due to their generalized pre-training objectives. Addressing this gap necessitates bridging the divide between broad cross-modal knowledge and the specific inference demands of VideoQA tasks. To this end, we introduce HeurVidQA, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning. By delivering fine-grained heuristics, we improve the model's ability to identify and interpret key entities and actions, thereby enhancing its reasoning capabilities. Extensive evaluations across multiple VideoQA datasets demonstrate that our method significantly outperforms existing models, underscoring the importance of integrating domain-specific knowledge into video-language models for more accurate and context-aware VideoQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。