arXiv:2607.00309cs.SDcs.CL2026-07中稿 · the International …

用自然语言实时生成可调控的音乐环境,让表演者像调色盘一样操控声音氛围。

A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models

论文配图:A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models
图 1 · 摘自论文原文
  • 通过语言指令生成可读配置,实现对声音氛围的精细控制
  • 采用检索、API和本地小模型三类后端,均输出统一声学结构
  • 支持无中断连续播放,即使大模型响应慢也能无缝衔接

我们提出一种实时音乐交互界面,将自然语言场景描述转化为动态演进的程序化音景。表演者输入如“午夜温暖的爵士咖啡馆”等提示,即可通过直接调节参数(如降低亮度、切换节奏风格)来引导声音变化,每次调整均产生可预测的听觉反馈,无需重新提示。与依赖GPU生成完整波形的文本到音频系统不同,本工具生成的是人类可读的配置,基于分类架构设计,大多数有效组合均保持音乐连贯性。提供三种可替换后端:嵌入检索实现亚秒级纯CPU运行,通过API调用托管大模型,以及一个微调的270M本地模型,三者输出相同配置结构。采用实时生成架构,在后台持续处理新指令的同时无缝播放音频;即使大模型响应需5-12秒,听众仍听不到中断,将文本到音乐转变为持续可表演的流式过程。我们使用LAION-CLAP在保留提示上评估文本-音频语义对齐,发现检索式配置优于随机有效配置,同时该指标也用于构建检索映射。报告性能观察、非正式听众反馈,并发布SDK、数据集、模型及音视频表演界面材料。

原文摘要 · Abstract (English)

We present a real-time musical interface that converts natural-language scene descriptions into evolving procedural soundscapes. A performer types a prompt such as "warm jazz cafe at midnight" and steers it through direct parameter adjustments - stepping brightness down, switching a rhythm style - each producing a predictable, audible shift without re-prompting. Where GPU-bound text-to-audio systems synthesize monolithic waveforms, our instrument generates human-readable configurations over a categorical schema, enabling fine-grained performer control; most valid combinations are designed to sound musically coherent. Three interchangeable backends - embedding retrieval for sub-second CPU-only use, hosted LLMs via API, and a fine-tuned 270M local model - all emit the same schema. A live generator architecture continuously emits audio while resolving new instructions in the background, crossfading seamlessly when ready; even when an LLM takes 5-12 seconds to respond, the audience hears uninterrupted sound - reframing text-to-music as an ongoing performable stream rather than a one-shot generation. We evaluate text-audio semantic alignment using LAION-CLAP on held-out prompts as a technical proxy, finding that retrieval-based configuration outperforms random valid configurations on this metric, while noting that LAION-CLAP also informed retrieval-map construction. We report performance observations, informal listener feedback, and release materials for the SDK, dataset artifacts, model, and audiovisual performance interface.

音景生成语言控制实时演奏程序化音频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。