arXiv:2506.16082cs.CV2025-06被引 1

通过注入位置与事件关系先验,提升视频密集描述的定位与生成质量

PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning

  • 用位置锚定查询提供场景化事件候选区域,减少无效提议
  • 引入事件关系编码器,增强事件间语义连贯性,提升描述质量
  • 在ActivityNet和YouCook2上表现优异,适合视频理解与生成任务

密集视频描述旨在定位并描述未剪辑视频中的多个事件。现有方法多采用基于Transformer的架构,端到端联合完成事件定位与描述生成,但依赖大量训练数据,且性能受限。本文提出新型框架PR-DETR,将显式的位置与关系先验注入检测Transformer,同时提升定位精度与描述质量。一方面,生成一组位置锚定查询,提供特定场景下的事件位置与语义信息,作为初始搜索区域,剔除不合理提议;另一方面,设计事件关系编码器,显式计算事件边界间的关系,引导事件交互,增强描述语义连贯性。大量消融实验验证了位置与关系先验的有效性。在ActivityNet Captions和YouCook2数据集上的实验结果表明,该方法具有竞争力。

原文摘要 · Abstract (English)

Dense video captioning is a challenging task that aims to localize and caption multiple events in an untrimmed video. Recent studies mainly follow the transformer-based architecture to jointly perform the two sub-tasks, i.e., event localization and caption generation, in an end-to-end manner. Based on the general philosophy of detection transformer, these methods implicitly learn the event locations and event semantics, which requires a large amount of training data and limits the model's performance in practice. In this paper, we propose a novel dense video captioning framework, named PR-DETR, which injects the explicit position and relation prior into the detection transformer to improve the localization accuracy and caption quality, simultaneously. On the one hand, we first generate a set of position-anchored queries to provide the scene-specific position and semantic information about potential events as position prior, which serves as the initial event search regions to eliminate the implausible event proposals. On the other hand, we further design an event relation encoder to explicitly calculate the relationship between event boundaries as relation prior to guide the event interaction to improve the semantic coherence of the captions. Extensive ablation studies are conducted to verify the effectiveness of the position and relation prior. Experimental results also show the competitive performance of our method on ActivityNet Captions and YouCook2 datasets.

视频描述注意力机制先验知识目标检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。