arXiv:2411.18180cs.CV2024-11CVPR被引 12

让电影字幕更独特,避免重复描述。

DistinctAD: Distinctive Audio Description Generation in Contexts

  • 分两阶段生成,先对齐视听模态,再减少上下文冗余。
  • 在多个评测集上优于基线,尤其在召回率指标上提升明显。
  • 适合需要高质量、不重复字幕的视频内容生成场景。

音频描述(AD)旨在以文本形式叙述电影中非对话内容,如角色、动作或场景设定。自动生成AD仍具挑战性,主要源于:一、电影AD数据与训练视觉语言模型的现有数据存在领域差距;二、长片中相邻视觉片段高度相似导致上下文冗余。本文提出DistinctAD,一种强调独特性的两阶段框架。为缓解领域差距,引入无需额外AD语料的CLIP-AD适配策略,实现全局与细粒度层面的模态对齐。第二阶段包含两项创新:(i) 上下文期望最大化注意力(EMA)模块,通过提取连续视频片段的共性基底来降低冗余;(ii) 显式独特词预测损失,过滤上下文中重复词汇,确保当前描述使用特定术语。在MAD-Eval、CMD-AD和TV-AD基准上的全面评估表明,DistinctAD持续优于基线模型,尤其在Recall@k/N指标上表现突出,验证了其生成高质量、独特化音频描述的有效性。

原文摘要 · Abstract (English)

Audio Descriptions (ADs) aim to provide a narration of a movie in text form, describing non-dialogue-related narratives, such as characters, actions, or scene establishment. Automatic generation of ADs remains challenging due to: i) the domain gap between movie-AD data and existing data used to train vision-language models, and ii) the issue of contextual redundancy arising from highly similar neighboring visual clips in a long movie. In this work, we propose DistinctAD, a novel two-stage framework for generating ADs that emphasize distinctiveness to produce better narratives. To address the domain gap, we introduce a CLIP-AD adaptation strategy that does not require additional AD corpora, enabling more effective alignment between movie and AD modalities at both global and fine-grained levels. In Stage-II, DistinctAD incorporates two key innovations: (i) a Contextual Expectation-Maximization Attention (EMA) module that reduces redundancy by extracting common bases from consecutive video clips, and (ii) an explicit distinctive word prediction loss that filters out repeated words in the context, ensuring the prediction of unique terms specific to the current AD. Comprehensive evaluations on MAD-Eval, CMD-AD, and TV-AD benchmarks demonstrate the superiority of DistinctAD, with the model consistently outperforming baselines, particularly in Recall@k/N, highlighting its effectiveness in producing high-quality, distinctive ADs.

音频描述去冗余多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。