用代码当桥梁，让文本模型生成音乐、声音和语音。
Probing Audio-Generation Capabilities of Text-Based Language Models
- 用代码作为中介，让文本模型生成音频
- 复杂度越高，生成效果越差，基础音符尚可
- 适合对多模态生成感兴趣的开发者
文本模型如何理解音频世界？本研究探究大语言模型（LLMs）在仅训练于文本数据的情况下，能否通过提示生成音频。采用三层递进方法：1）乐谱生成，2）环境声，3）人声。为弥合文本与音频的鸿沟，利用代码作为中间媒介，引导模型输出可执行代码以生成目标音频。使用FAD和CLAP分数评估生成音频的质量与准确性。结果表明，尽管模型能生成基本音频特征，但随着音频复杂度提升，性能显著下降。这说明虽然模型具备对听觉世界的潜在理解，但将其转化为实际音频输出的能力仍十分初级。未来改进生成技术，有望提升文本型模型在音频生成方面的表现。
原文摘要 · Abstract (English)
How does textual representation of audio relate to the Large Language Model's (LLMs) learning about the audio world? This research investigates the extent to which LLMs can be prompted to generate audio, despite their primary training in textual data. We employ a three-tier approach, progressively increasing the complexity of audio generation: 1) Musical Notes, 2) Environmental Sounds, and 3) Human Speech. To bridge the gap between text and audio, we leverage code as an intermediary, prompting LLMs to generate code that, when executed, produces the desired audio output. To evaluate the quality and accuracy of the generated audio, we employ FAD and CLAP scores. Our findings reveal that while LLMs can generate basic audio features, their performance deteriorates as the complexity of the audio increases. This suggests that while LLMs possess a latent understanding of the auditory world, their ability to translate this understanding into tangible audio output remains rudimentary. Further research into techniques that can enhance the quality and diversity of LLM-generated audio can lead to an improvement in the performance of text-based LLMs in generating audio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。