用连续空间融合语音与文本，打造首个希腊语语音大模型
VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion
- 通过中间文本空间对齐语音与语言模型表示
- 希腊语语音识别达到领先水平，相对提升约20%
- 支持离线与流式处理，适合多语言低资源场景
我们提出一种多模态融合框架，连接基于解码器的预训练大语言模型（LLM）与Whisper等声学编码器-解码器架构，旨在构建具备语音能力的大语言模型。不同于直接使用音频嵌入，我们探索了一种中间音频条件文本空间，作为更有效的对齐机制。方法在连续文本表示空间中运行，通过跨模态注意力将Whisper的隐藏解码状态与LLM状态融合，支持离线和流式模式。我们引入了首个希腊语语音大模型VoxKrikri，分析表明该方法能有效对齐多模态表示。结果表明，连续空间融合是构建多语言及低资源语音大模型的有前景路径，同时在希腊语自动语音识别任务上取得最佳表现，各基准平均相对提升约20%。
原文摘要 · Abstract (English)
We present a multimodal fusion framework that bridges pre-trained decoder-based large language models (LLM) and acoustic encoder-decoder architectures such as Whisper, with the aim of building speech-enabled LLMs. Instead of directly using audio embeddings, we explore an intermediate audio-conditioned text space as a more effective mechanism for alignment. Our method operates fully in continuous text representation spaces, fusing Whisper's hidden decoder states with those of an LLM through cross-modal attention, and supports both offline and streaming modes. We introduce \textit{VoxKrikri}, the first Greek speech LLM, and show through analysis that our approach effectively aligns representations across modalities. These results highlight continuous space fusion as a promising path for multilingual and low-resource speech LLMs, while achieving state-of-the-art results for Automatic Speech Recognition in Greek, providing an average $\sim20\%$ relative improvement across benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。