用专家路由机制提升函数式编程代码生成效果
FPMoE: A Sparse Mixture-of-Experts Approach to Functional Code Generation
- 设计三语言专用专家+共享专家的稀疏混合专家架构
- 仅30亿活跃参数就达到大模型水平，超越多个14B以上模型
- 适合需要高效生成Haskell/OCaml/Scala代码的研究者与开发者
尽管基于大语言模型的代码生成进展迅速，现有模型主要针对命令式语言训练，对函数式编程语言（如Haskell、OCaml、Scala）仍严重缺乏支持，甚至前沿模型在这些语言上表现显著下降。微调虽是自然解决方案，但单语言微调无法捕捉共通的函数式抽象，而多语言合并微调又引入跨语言干扰。为此，我们提出FPMoE，一个轻量级开源代码生成模型，采用稀疏混合专家（MoE）架构，包含三个语言专用专家（分别对应Haskell、OCaml、Scala）和一个共享专家，用于捕捉跨语言的函数式模式（如单子推理、类型导向编程）。该设计同时解决两类问题：专用专家消除干扰，共享专家保留被忽略的通用抽象。在FPEval测试集上，FPMoE显著优于微调基线，在仅30亿活跃参数的情况下，性能媲美DeepSeek-Coder-6.7B、Qwen2.5-Coder-14B-Instruct及Qwen3-Coder-30B-A3B等更大模型。
原文摘要 · Abstract (English)
Despite rapid progress in LLM-based code generation, existing models are predominantly trained on imperative languages, leaving functional programming languages (FPLs) such as Haskell, OCaml, and Scala chronically underexplored, with even frontier models performing substantially worse on FPLs. Fine-tuning is a natural remedy, but our experiments show that per-language fine-tuning fails to capture shared functional abstractions, while merged multi-language fine-tuning introduces cross-language interference. To address this, we introduce FPMoE, a lightweight, open-source code generation model built on a sparse Mixture-of-Experts (MoE) architecture with three language-specific routed experts (one each for Haskell, OCaml, and Scala) and a shared expert that captures cross-language functional patterns such as monadic reasoning and type-directed programming. This design resolves both failure modes simultaneously: dedicated experts eliminate interference, while the shared expert preserves abstractions that per-language models miss. On FPEval, FPMoE substantially outperforms fine-tuned baselines and, with only 3B active parameters, matches the performance of much larger models including DeepSeek-Coder-6.7B, Qwen2.5-Coder-14B-Instruct, and Qwen3-Coder-30B-A3B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。