arXiv:2501.03183cs.CLcs.AI2025-01

用分类器引导生成跨模态描述,无需训练即可提升音频等场景的生成质量。

Classifier-Guided Captioning Across Modalities

  • 通过冻结语言模型+文本分类器在推理时引导生成
  • 结合GPT-4自动生成数据,使零样本音频描述性能达新高
  • 适用于音频、视频等多模态场景,部署简单无须微调

现有字幕系统依赖特定场景(如图像标注)的数据训练,难以泛化到音频或视频等其他模态。本文提出一种跨模态字幕生成框架:包含一个冻结的语言模型和一个在GPT-4生成数据上训练的文本分类器,该分类器通过定制提示增强生成内容的语义特性,仅在推理阶段使用,无需对基础模型进行再训练。实验表明,该框架可显著提升零样本音频字幕质量,在多个模型与模态上表现优异,尤其在音频场景下达到当前最优水平。

原文摘要 · Abstract (English)

Most current captioning systems use language models trained on data from specific settings, such as image-based captioning via Amazon Mechanical Turk, limiting their ability to generalize to other modality distributions and contexts. This limitation hinders performance in tasks like audio or video captioning, where different semantic cues are needed. Addressing this challenge is crucial for creating more adaptable and versatile captioning frameworks applicable across diverse real-world contexts. In this work, we introduce a method to adapt captioning networks to the semantics of alternative settings, such as capturing audibility in audio captioning, where it is crucial to describe sounds and their sources. Our framework consists of two main components: (i) a frozen captioning system incorporating a language model (LM), and (ii) a text classifier that guides the captioning system. The classifier is trained on a dataset automatically generated by GPT-4, using tailored prompts specifically designed to enhance key aspects of the generated captions. Importantly, the framework operates solely during inference, eliminating the need for further training of the underlying captioning model. We evaluate the framework on various models and modalities, with a focus on audio captioning, and report promising results. Notably, when combined with an existing zero-shot audio captioning system, our framework improves its quality and sets state-of-the-art performance in zero-shot audio captioning.

跨模态音频字幕零样本生成引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。