用AudioSet关键词增强音频描述生成，更准且省资源。
Enhancing Audio Captioning with Auxiliary AudioSet Semantics
- 用ConvNeXt提取音频特征，加Top-K AudioSet关键词提供语义提示。
- 六层BART结构在Clotho V2和AudioCaps上达主流水平，不依赖大模型。
- 适合资源受限场景，兼顾准确性和推理效率。
自动音频描述（AAC）旨在为复杂声学场景生成自然语言描述，连接听觉感知与语言理解。然而，词汇选择不确定性和对大规模序列到序列或大语言模型（LLM）的依赖限制了实际部署。本文提出一种资源高效的AAC框架，显式利用辅助AudioSet语义来锚定描述生成。使用ConvNeXt编码器提取帧级音频表示，并叠加预测出的Top-$K$ AudioSet关键词，为解码提供结构化上下文线索。一个紧凑的六层BART-style解码器基于此联合声学-语义表示进行生成，无需大模型级别的解码。所提设计在紧凑架构内平衡了语义锚定与计算效率。在Clotho V2和AudioCaps数据集上的评估表明，在实际部署约束下仍能实现具有竞争力的描述质量。
原文摘要 · Abstract (English)
Automatic Audio Captioning (AAC) seeks to generate natural language descriptions of complex acoustic scenes, bridging auditory perception and language understanding. However, word-selection indeterminacy and increasing reliance on large-scale sequence-to-sequence or LLM-based models limit practical deployment. We propose a resource-efficient AAC framework that explicitly grounds caption generation in auxiliary AudioSet semantics. Frame-level acoustic representations extracted using a ConvNeXt encoder are augmented with top-$K$ predicted AudioSet keywords, providing structured contextual cues for decoding. A compact six-layer BART-style decoder conditions on this joint acoustic-semantic representation, enabling caption generation without LLM-scale decoding. The proposed design balances semantic grounding and computational efficiency within a compact architecture. Evaluations on Clotho V2 and AudioCaps confirm competitive caption quality under practical deployment constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。