用专家混合模型统一纠正语音、文本、图像转文字的错误，效果优于大模型。
NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model
- 采用多任务专家混合架构，自动将不同数据集的输入路由到对应专家处理
- 在开放语音识别榜单上平均降低5.0%词错误率，翻译任务BLEU显著提升
- 零样本测试中比GPT-3.5和Claude-Opus低15.5%-27.6%词错误率，适合多模态纠错场景
构建通用后识别错误纠正器的关键挑战在于：如何高效地在大规模跨域数据集上训练模型？答案在于学习各数据集的特异性特征，并在单一模型中融合其知识。以往方法通过为每个任务单独部署纠正语言模型实现，导致参数量大幅增加。本文提出多任务纠正型专家混合模型（Multi-Task Correction MoE），让专家成为语音转文本、语言转文本及视觉转文本数据集的专属处理单元，通过学习将各数据集的词元路由至对应专家。在Open ASR Leaderboard上的实验表明，该方法实现了新的最先进性能，平均相对词错误率（WER）降低5.0%，且在语音与翻译任务中显著提升BLEU分数。在零样本评估中，NeKo在Hyporadise基准上相比GPT-3.5和Claude-Opus实现15.5%至27.6%的相对词错误率降低，同时在语法纠错与后OCR修正任务中表现优异，展现强大的多任务能力。
原文摘要 · Abstract (English)
Construction of a general-purpose post-recognition error corrector poses a crucial question: how can we most effectively train a model on a large mixture of domain datasets? The answer would lie in learning dataset-specific features and digesting their knowledge in a single model. Previous methods achieve this by having separate correction language models, resulting in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose a Multi-Task Correction MoE, where we train the experts to become an ``expert'' of speech-to-text, language-to-text and vision-to-text datasets by learning to route each dataset's tokens to its mapped expert. Experiments on the Open ASR Leaderboard show that we explore a new state-of-the-art performance by achieving an average relative 5.0% WER reduction and substantial improvements in BLEU scores for speech and translation tasks. On zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to 27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。