用专家混合与高质量数据，打造高效精准的代码大模型。
Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM
- 采用专家混合架构+程序分析驱动的数据筛选方法。
- 性能持平顶尖7B模型，部署资源减少50%无损性能。
- 适合追求效率与精度平衡的代码生成研究者使用。
近期代码大语言模型在代码生成与理解方面展现卓越能力，但如何在保证全面性能的同时实现极致效率仍具挑战。本文提出Ling-Coder-Lite，通过高效的Mixture-of-Experts（MoE）架构结合基于程序分析的高质量数据筛选方法，构建兼具高效性与强大性能的代码大模型。在12个代表性编码基准上，其性能与Qwen2.5-Coder-7B、DeepSeek-Coder-V2-Lite等同规模先进模型相当，同时具备更优的延迟与吞吐表现。实际部署中，相比同规模稠密模型，资源消耗降低50%且性能无损失。为推动该领域研究，我们开源了模型及训练阶段的高质量数据集，可通过https://huggingface.co/inclusionAI/Ling-Coder-lite 获取。
原文摘要 · Abstract (English)
Recent advancements in code large language models (LLMs) have demonstrated remarkable capabilities in code generation and understanding. It is still challenging to build a code LLM with comprehensive performance yet ultimate efficiency. Many attempts have been released in the open source community to break the trade-off between performance and efficiency, such as the Qwen Coder series and the DeepSeek Coder series. This paper introduces yet another attempt in this area, namely Ling-Coder-Lite. We leverage the efficient Mixture-of-Experts (MoE) architecture along with a set of high-quality data curation methods (especially those based on program analytics) to build an efficient yet powerful code LLM. Ling-Coder-Lite exhibits on-par performance on 12 representative coding benchmarks compared to state-of-the-art models of similar size, such as Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite, while offering competitive latency and throughput. In practice, we achieve a 50\% reduction in deployment resources compared to the similar-sized dense model without performance loss. To facilitate further research and development in this area, we open-source our models as well as a substantial portion of high-quality data for the annealing and post-training stages. The models and data can be accessed at~\url{https://huggingface.co/inclusionAI/Ling-Coder-lite}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。