让文字生成带空间感的多声源立体音频，提升沉浸体验。
TTMBA: Towards Text To Multiple Sources Binaural Audio Generation
- 用大语言模型解析文本中的时间与位置信息，结构化生成声事件
- 分步生成多路单声道音频，再转为立体声，实现精准空间定位
- 适合需要高沉浸感音频场景的应用，如虚拟现实与游戏
现有文本到音频（TTA）生成方法大多产生单声道输出，忽略了沉浸式听觉体验所需的空间信息。为解决此问题，我们提出一种级联式文本到多声源立体音频生成方法（TTMBA），实现时间和空间双重控制。首先，利用预训练大语言模型（LLM）将文本分解为包含每个声音事件的时间与空间细节的结构化格式；其次，通过预训练的单声道音频生成网络为每个事件生成不同持续时间的单声道音频；接着，基于来自LLM的空间数据，使用基于神经网络的立体声渲染模块将这些单声道音频转换为立体音频；最后，按各音频的起始时间进行排序，生成多声源立体音频。实验结果表明，该方法在音频生成质量与空间感知准确性方面均表现更优。
原文摘要 · Abstract (English)
Most existing text-to-audio (TTA) generation methods produce mono outputs, neglecting essential spatial information for immersive auditory experiences. To address this issue, we propose a cascaded method for text-to-multisource binaural audio generation (TTMBA) with both temporal and spatial control. First, a pretrained large language model (LLM) segments the text into a structured format with time and spatial details for each sound event. Next, a pretrained mono audio generation network creates multiple mono audios with varying durations for each event. These mono audios are transformed into binaural audios using a binaural rendering neural network based on spatial data from the LLM. Finally, the binaural audios are arranged by their start times, resulting in multisource binaural audio. Experimental results demonstrate the superiority of the proposed method in terms of both audio generation quality and spatial perceptual accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。