arXiv:2505.14173cs.CL2025-05ACL被引 2

提出分层任务引导与上下文感知路由,提升翻译模型专家选择精准度。

THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation

  • 分层预测领域/语言标签,动态分配任务级专家
  • 引入全局上下文信息,提升令牌路由准确率
  • 兼容主流路由策略,参数激活率低且性能优

稀疏混合专家(MoE)在神经机器翻译中取得显著进展,但现有方法存在两大局限:一是直接使用任务知识(如领域/语言特异性知识),而这些知识在实际应用中通常不可用,且忽略了自然存在的领域/语言分组特性;二是专家选择仅依赖局部词元表示,未考虑全局上下文信息。为此,本文提出THOR-MoE,通过分层任务引导与上下文感知路由机制解决上述问题。首先,预测领域/语言标签并提取混合表示,以分层方式分配任务级专家;其次,在预选的任务级专家集合中注入上下文信息,增强词元路由精度,使每个词元能被更专业、更合适的专家处理。在多领域与多语言翻译基准上,不同架构的实验均验证了其优越性。例如,相比原始Top-p路由,THOR-MoE在多领域任务中实现0.75 BLEU平均提升,且激活参数少于22%。

原文摘要 · Abstract (English)

The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT). However, there exist two limitations in current MoE solutions which may lead to sub-optimal performance: 1) they directly use the task knowledge of NMT into MoE (\emph{e.g.}, domain/linguistics-specific knowledge), which are generally unavailable at practical application and neglect the naturally grouped domain/linguistic properties; 2) the expert selection only depends on the localized token representation without considering the context, which fully grasps the state of each token in a global view. To address the above limitations, we propose THOR-MoE via arming the MoE with hierarchical task-guided and context-responsive routing policies. Specifically, it 1) firstly predicts the domain/language label and then extracts mixed domain/language representation to allocate task-level experts in a hierarchical manner; 2) injects the context information to enhance the token routing from the pre-selected task-level experts set, which can help each token to be accurately routed to more specialized and suitable experts. Extensive experiments on multi-domain translation and multilingual translation benchmarks with different architectures consistently demonstrate the superior performance of THOR-MoE. Additionally, the THOR-MoE operates as a plug-and-play module compatible with existing Top-$k$~\cite{shazeer2017} and Top-$p$~\cite{huang-etal-2024-harder} routing schemes, ensuring broad applicability across diverse MoE architectures. For instance, compared with vanilla Top-$p$~\cite{huang-etal-2024-harder} routing, the context-aware manner can achieve an average improvement of 0.75 BLEU with less than 22\% activated parameters on multi-domain translation tasks.

机器翻译混合专家路由机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。