提出新方法自动识别电影中何时何地生成语音描述，提升视障用户观影体验。
From Visual Cues to Spoken Narration: Rethinking Audio Description
- 分两阶段定位视觉线索与配音时机，联合预测讲述内容和插入位置。
- 在长达8分钟的长视频上，平均mAP比基线高5到12个百分点。
- 首个针对长片段多段语音描述的基准数据集，适合无障碍技术研究者。
音频描述（AD）在对话间隙中对视觉事件进行语音解说，使电影对视障观众可访问。该任务需同时确定讲述内容（哪个视觉事件）和插入时机（何时插入），以获得最佳用户体验。以往工作多将问题简化为预分割视频片段的视频字幕生成，即内容基本预设、时机被完全忽略。本文提出Cue2Narrate，一种两阶段流水线，联合预测长未修剪电影片段中的讲述内容与插入时机。双头音视频定位器为每段AD生成两个时间上分离的窗口：视觉线索窗口与语音解说窗口。基于LoRA微调的视觉语言模型从预测的视觉证据生成简洁的音频描述，使用描述排序损失训练，使相同帧的负样本描述得分低于真实描述。为评估这一新问题，我们引入LongLSMDC基准，包含最长8分钟、平均约6.5分钟的电影片段。在LongLSMDC上，Cue2Narrate相比仅视频或仅音频定位基线，平均mAP提升5–12点。在预测窗口和真实窗口两种评估下，其生成的音频描述均优于对应微调的基线视觉语言模型。这些结果建立了首个针对长片段多段音频描述生成的基准。数据与代码：https://github.com/multimodal-ai-lab/Cue2Narrate
原文摘要 · Abstract (English)
Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The problem requires determining both what (which visual event) and when (position for inserting the AD) to narrate, to achieve the best user experience. Prior work has largely reduced the problem to video captioning of pre-segmented video clips, i.e., what is largely predefined and when is ignored entirely. We propose Cue2Narrate, a two-stage pipeline that jointly predicts what and when to narrate in longer untrimmed movie clips. A dual-head audio-visual localizer predicts two temporally distinct windows per AD utterance: a visual cue window and a spoken narration window. A LoRA-adapted VLM then generates concise ADs from the predicted visual evidence, trained with a Description Ranking Loss that ranks captions (negative samples) of the same frames lower than the GT AD. To benchmark this new problem statement, we introduce the LongLSMDC benchmark with up to 8-min movie clips (~6.5min on average). On LongLSMDC, Cue2Narrate outperforms video-only and audio-only localization baselines by 5--12 points in avg. mAP. Under both predicted- and GT-window evaluation, Cue2Narrate improves AD generation over the corresponding fine-tuned base VLM. These results establish the first benchmark for multi-segment AD generation on long-form clips. Data & Code: https://github.com/multimodal-ai-lab/Cue2Narrate
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。