不依赖音频编码器的音乐描述模型，用知识蒸馏提升生成效果
CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning
- 将教师模型的知识分路径注入投影器和语言模型，而非仅传给语言模型
- 在AudioCaps上比纯语言模型蒸馏提升12.18点，接近保留编码器的上限
- 仅需13.2M参数投影器，推理时无编码器开销，适合低延迟场景
当前自动音频描述系统通常将冻结的音频编码器与大型语言模型（LLM）通过可训练投影器结合，导致推理成本高且受固定声学特征限制。我们提出CARD，一种无需编码器的音频描述模型：13.2M参数的投影器通过融合的LoRA适配器向冻结的LLM输入信息，而训练用的教师模型在推理时被丢弃。CARD将预训练音频教师模型（CLAP-HTSAT）的知识蒸馏到系统中，但不是仅注入语言模型，而是将感知阶段特征传给投影器、语义阶段特征传给语言模型。该设计使AudioCaps上的CIDEr-D提升12.18点，Clotho上提升5.21点，达到55.4，而保留编码器的上限为66.4，证明知识放置位置与存在同等重要。
原文摘要 · Abstract (English)
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。