用混合专家模型提升大模型多语言编程能力
MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts
- 设计双层混合专家结构，分词级和段级优化专家选择
- 在12种编程语言上实现平均7.8%的生成准确率提升
- 适合需要高效多语言代码生成的开发者与研究者
尽管大语言模型具备出色的代码生成能力，但多语言代码生成仍极具挑战。为解决此问题，我们旨在以有限计算资源提升基础大模型的多编程语言（MultiPL）性能。将多编程语言视为多种自然语言的特例，提出一种基于混合专家（MoE）的MultiPL扩展方法——MultiPL-MoE。该方法结合两个配对的MoE，在词元与段落两级优化专家选择：词元级MoE采用共享专家与新型门控权重归一化，便于与段落级结果融合；段落级MoE引入滑动窗口划分输入序列，并采用专家选段路由策略，使专家可选择前k个相关段落。实验验证了MultiPL-MoE的有效性。
原文摘要 · Abstract (English)
Despite LLMs' excellent code creation capabilities, multilingual code generation remains extremely challenging. To address this, we intent to improve the multi-programming-lingual (MultiPL) performance of the base LLMs while retaining the most popular ones using restricted computational resources. We consider MultiPL to be a special case of multiple natural languages and propose a MultiPL extension of LLMs utilizing a hybrid mixture of experts (MoE), called MultiPL-MoE. Specifically, MultiPL-MoE combines two paired MoEs to optimize expert selection at both the token and segment levels. The token-level MoE is a standard upcycling MoE structure with a shared expert and a novel gate weight normalization approach that aids in the final fusion with the segment-level MoE. The segment-level MoE incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages: First, using a sliding window to partition the input token sequence into multiple segments; Then, adopting an expert-choice routing strategy that allows experts to select the top-k segments. The results of the experiment proved the effectiveness of MultiPL-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。