Ichigo实现语音与文本混合输入的实时对话，延迟仅111毫秒。
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant
- 将语音转为离散符号，与文本统一用Transformer处理
- 在多语言语音数据上预训练，指令数据微调后性能领先
- 适合追求低延迟的语音助手开发，支持开源协作
大型语言模型（LLM）已革新自然语言处理，但语音任务中音视频模态融合仍具挑战。本文提出Ichigo，一种混合模态模型，可无缝处理语音与文本交错序列。通过分词化的早期融合方法，将语音量化为离散标记，并采用统一的Transformer架构处理语音与文本，实现跨模态联合推理与生成，无需额外适配器。我们设计了完整的训练流程，包括在多语言语音识别数据集上预训练，以及在精心筛选的指令数据集上微调。Ichigo在语音问答基准上表现达到顶尖水平，优于现有开源语音语言模型，且与级联系统相当。尤为突出的是，其首字生成延迟仅为111毫秒，显著低于当前模型。本方法不仅推动多模态人工智能发展，也为小型研究团队参与开源语音-语言模型贡献提供了有效框架。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have revolutionized natural language processing, but their application to speech-based tasks remains challenging due to the complexities of integrating audio and text modalities. This paper introduces Ichigo, a mixed-modal model that seamlessly processes interleaved sequences of speech and text. Utilizing a tokenized early-fusion approach, Ichigo quantizes speech into discrete tokens and employs a uniform transformer-based architecture for both speech and text modalities. This method enables joint reasoning and generation across modalities without the need for separate adapters. We present a comprehensive training methodology, including pre-training on multilingual speech recognition datasets and fine-tuning on a curated instruction dataset. Ichigo demonstrates state-of-the-art performance on speech question-answering benchmarks, outperforming existing open-source speech language models and achieving comparable results to cascaded systems. Notably, Ichigo exhibits a latency of just 111 ms to first token generation, significantly lower than current models. Our approach not only advances the field of multimodal AI but also provides a framework for smaller research teams to contribute effectively to open-source speech-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。