将音频转为极低码率离散标记，让大模型能像处理文本一样理解生成声音。
Make Some Noise: Towards LLM audio reasoning and generation using sound tokens
- 用变分量化+条件流匹配，把音频转成0.23kbps离散符号
- 模型在音频理解上媲美顶尖方法，但生成效果不佳
- 适合想探索声音与语言融合的多模态研究者
由于音频的连续性及高采样率，将其融入大语言模型（LLM）仍具挑战。本文提出一种新方法，结合变分量化与条件流匹配，将音频转换为0.23kbps的超低码率离散标记，实现与文本标记的无缝集成。我们采用低秩适配（LoRA）微调预训练文本模型，评估其在音频理解与生成方面的多模态能力。所提分词器在多个包含多样化声学事件的数据集上优于传统VQ-VAE。尽管音色细节大量丢失，使用离散标记训练的多模态LLM在音频理解任务中表现接近当前最优水平，但音频生成性能较差。结果表明，亟需更大更丰富的数据集和更优评估指标以推动多模态LLM发展。
原文摘要 · Abstract (English)
Integrating audio comprehension and generation into large language models (LLMs) remains challenging due to the continuous nature of audio and the resulting high sampling rates. Here, we introduce a novel approach that combines Variational Quantization with Conditional Flow Matching to convert audio into ultra-low bitrate discrete tokens of 0.23kpbs, allowing for seamless integration with text tokens in LLMs. We fine-tuned a pretrained text-based LLM using Low-Rank Adaptation (LoRA) to assess its effectiveness in achieving true multimodal capabilities, i.e., audio comprehension and generation. Our tokenizer outperforms a traditional VQ-VAE across various datasets with diverse acoustic events. Despite the substantial loss of fine-grained details through audio tokenization, our multimodal LLM trained with discrete tokens achieves competitive results in audio comprehension with state-of-the-art methods, though audio generation is poor. Our results highlight the need for larger, more diverse datasets and improved evaluation metrics to advance multimodal LLM performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。