一个能听会说的智能助手模型，一次处理语音与文本，性能优于专用模型。
SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions
- 融合语音与文本输入，端到端完成多任务交互
- 语音触发检测错误率降低64%，设备定向语音识别提升22%
- 适合构建轻量级、高效率的虚拟助手系统
本文提出并评估了SELMA，一种用于虚拟助手交互的语音增强型语言模型，将音频与文本作为输入整合至大型语言模型（LLM）中。SELMA旨在单一端到端模型内同时处理三项主要任务和两项辅助任务。我们采用低秩适配模块实现音频编码器与LLM的参数高效训练，并引入特征池化策略，使系统能够识别全局模式，提升对不依赖单个序列元素的任务的准确性。在语音触发（VT）检测、设备定向语音检测（DDSD）和自动语音识别（ASR）上的实验表明，该方法显著简化了虚拟助手的传统输入处理流程，且性能优于各任务专用模型。SELMA在VT检测任务上实现相对等错误率改善64%，在DDSD任务上改善22%，同时词错误率接近基线水平。
原文摘要 · Abstract (English)
In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。