arXiv:2606.21453cs.HCcs.AI2026-06

用文本监督微调语音模型,让语音助手无需标注语料也能准确理解指令。

CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents

  • 仅用文本任务标注微调语音模型,实现语音输入到结构化输出的直接生成。
  • 在噪声环境下表现更优,尤其能更好保留任务语义。
  • 适合缺乏配对语音-目标数据的语音助手快速适配场景。

任务导向的语音助手需将用户口语请求映射为语义框架、可执行动作或函数调用。传统方法是级联语音识别(ASR)与文本大模型,但识别错误会传播至下游结构化输出生成,尤其在嘈杂环境下。语音语言模型(SLMs)提供直接基于语音的替代方案,但通常需成对的语音-目标标注进行适配。为此,我们提出CORTIS,一种仅用文本形式任务监督微调语音模型的框架,使推理时无需特定任务的语音-目标标注即可生成结构化输出。我们在两个Qwen2.5-Omni骨干模型及三个任务导向语音数据集(含内部产品数据集)上评估CORTIS,结果表明其性能与使用相同文本监督训练的匹配级联模型相当,并在声学退化条件下优势更明显,尤其在保持高层任务语义方面。这表明当配对语音-目标数据难以获取时,仅用文本微调语音模型是一种可行的实用适配策略。

原文摘要 · Abstract (English)

Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls. A common approach is to cascade ASR with a text-based LLM, but transcription errors can propagate to downstream structured output generation, especially under noisy conditions. Spoken language models (SLMs) offer a direct speech-based alternative, yet adapting them to new tasks typically requires paired speech-target annotations. Motivated by this gap, we present CORTIS, a text-only adaptation framework for task-oriented voice agents. CORTIS fine-tunes SLMs using text-form task supervision, enabling speech-based structured output generation at inference time without task-specific speech-target annotations during adaptation. We evaluate CORTIS on two Qwen2.5-Omni backbones and three task-oriented speech datasets, including an in-house product dataset, and compare it with matched ASR-LLM cascades trained with the same text-form task supervision. Results show that CORTIS performs competitively with matched cascades and offers clearer advantages under acoustic degradation, particularly in preserving high-level task semantics. These findings suggest that text-only fine-tuning of SLMs can serve as a practical adaptation strategy for voice agents when paired speech-target data are costly to collect.

语音模型文本微调语音助手任务导向

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。