用显著性监督提升视频片段分割精度，让生成更准。
Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning
- 通过真值标注训练显著性模块，指导时间分割。
- 在YouCook2和ViTT上多项指标达顶尖水平。
- 适合需要精准事件定位的视频理解任务。
现有的检索增强型密集视频字幕（DVC）方法常因依赖启发式策略而无法准确对齐真实事件边界。本文提出框架STaRC，通过突出显示检测模块以监督帧级显著性，该模块使用来自DVC真值标注的二值标签进行训练，无需额外标注。我们进一步将显著性分数作为统一的时间信号，用于引导基于显著性的分割，并通过显式显著性提示注入解码器，指导字幕生成。通过施加显著性约束的分割，方法生成的时间一致片段能紧密对齐实际事件转换，从而提升检索准确性和上下文相关的字幕生成质量。我们在YouCook2和ViTT基准上进行了全面评估，STaRC在多数指标上达到当前最优性能。代码已开源。
原文摘要 · Abstract (English)
Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, \textbf{STaRC}, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。