70亿参数音频模型直接处理语音输入输出,支持对话与全双工交互。
Covo-Audio Technical Report
- 统一架构端到端处理连续音频,无需分步转换
- 在多任务上表现超越同类开源模型,对话理解能力强
- 提出智能与语音分离策略,灵活换声且不损对话能力
本文介绍 Covo-Audio,一个 70 亿参数的端到端 LALM,可直接处理连续音频输入并生成音频输出,采用大规模精选预训练和针对性后训练,在语音-文本建模、语音对话、语音理解、音频理解及全双工语音交互等广泛任务中达到当前同类规模模型的领先或竞争水平。大量评估显示,预训练基础模型在多个基准上展现出强大的语音-文本理解与语义推理能力,优于同规模代表性开源模型。Covo-Audio-Chat 作为对话优化版本,表现出色的口语对话能力,包括理解、上下文推理、指令遵循以及生成情境恰当且富有同理心的回应,验证其在真实对话助手场景中的适用性。Covo-Audio-Chat-FD 进化版在语音对话能力和全双工交互行为上均实现显著提升,展现实际鲁棒性。为降低端到端 LALM 在自然对话系统中的部署成本,我们提出智能-发音者解耦策略,将对话智能与语音合成分离,仅需少量文本转语音数据即可实现灵活语音定制,同时保持对话性能。整体结果表明,70 亿参数规模模型具备融合复杂音频智能与高层语义推理的强大潜力,并指明了构建更强大、更通用的 LALM 的可扩展路径。
原文摘要 · Abstract (English)
In this work, we present Covo-Audio, a 7B-parameter end-to-end LALM that directly processes continuous audio inputs and generates audio outputs within a single unified architecture. Through large-scale curated pretraining and targeted post-training, Covo-Audio achieves state-of-the-art or competitive performance among models of comparable scale across a broad spectrum of tasks, including speech-text modeling, spoken dialogue, speech understanding, audio understanding, and full-duplex voice interaction. Extensive evaluations demonstrate that the pretrained foundation model exhibits strong speech-text comprehension and semantic reasoning capabilities on multiple benchmarks, outperforming representative open-source models of comparable scale. Furthermore, Covo-Audio-Chat, the dialogue-oriented variant, demonstrates strong spoken conversational abilities, including understanding, contextual reasoning, instruction following, and generating contextually appropriate and empathetic responses, validating its applicability to real-world conversational assistant scenarios. Covo-Audio-Chat-FD, the evolved full-duplex model, achieves substantially superior performance on both spoken dialogue capabilities and full-duplex interaction behaviors, demonstrating its competence in practical robustness. To mitigate the high cost of deploying end-to-end LALMs for natural conversational systems, we propose an intelligence-speaker decoupling strategy that separates dialogue intelligence from voice rendering, enabling flexible voice customization with minimal text-to-speech (TTS) data while preserving dialogue performance. Overall, our results highlight the strong potential of 7B-scale models to integrate sophisticated audio intelligence with high-level semantic reasoning, and suggest a scalable path toward more capable and versatile LALMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。