用专家混合架构让模型高效支持多语言编程，兼顾性能与省资源。
Mix-of-Language-Experts Architecture for Multilingual Programming
- 采用共享+语言专用LoRA模块，实现知识共享与个性化适配。
- 参数量仅需单个模型的60%左右，准确率却显著高于通用模型。
- 适合需要低成本部署多语言代码生成系统的开发者使用。
大型语言模型在代码理解、生成和翻译等开发任务中表现优异。支持多语言编程通常有两种方式：一是对单一模型进行全语言微调，虽成本低但缺乏语言特异性；二是为每种语言单独微调模型，虽可专精但计算和存储开销大。本文提出MoLE（Mix-of-Language-Experts）架构，由基础模型、共享LoRA模块和多个语言专属LoRA模块组成，联合优化以实现跨语言知识共享与特化。推理时，模型根据代码片段自动路由至对应语言的LoRA模块。实验表明，MoLE在参数效率上优于独立微调各语言的LoRA方案，同时在准确性上超越统一微调的单一模型。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated impressive capabilities in aiding developers with tasks like code comprehension, generation, and translation. Supporting multilingual programming -- i.e., coding tasks across multiple programming languages -- typically requires either (1) finetuning a single LLM across all programming languages, which is cost-efficient but sacrifices language-specific specialization and performance, or (2) finetuning separate LLMs for each programming language, which allows for specialization but is computationally expensive and storage-intensive due to the duplication of parameters. This paper introduces MoLE (Mix-of-Language-Experts), a novel architecture that balances efficiency and specialization for multilingual programming. MoLE is composed of a base model, a shared LoRA (low-rank adaptation) module, and a collection of language-specific LoRA modules. These modules are jointly optimized during the finetuning process, enabling effective knowledge sharing and specialization across programming languages. During inference, MoLE automatically routes to the language-specific LoRA module corresponding to the programming language of the code token being generated. Our experiments demonstrate that MoLE achieves greater parameter efficiency compared to training separate language-specific LoRAs, while outperforming a single shared LLM finetuned for all programming languages in terms of accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。