arXiv:2603.16526cs.AI2026-03

用合成数据微调小模型,提升特定领域代码生成能力。

Exploring different approaches to customize language models for domain-specific text-to-code generation

  • 构建三个领域合成数据集,用于定制化代码生成。
  • LoRA微调在准确率和领域契合度上均优于提示与检索方法。
  • 适合需要低成本高精度代码生成的开发者使用。

大型语言模型在从自然语言生成可执行代码方面表现出色,但在需使用领域专用库、API或规范的编程场景中表现不佳。定制小型开源模型是替代大体量专有系统的一种成本效益更高的方案。本文研究如何利用合成数据将小型语言模型适配于特定领域代码生成任务。我们构建了三个基于Python生态系统的编程练习数据集:通用Python编程、Scikit-learn机器学习流程以及基于OpenCV的计算机视觉任务。基于这些数据集,评估三种定制策略:少样本提示(few-shot prompting)、检索增强生成(RAG)以及基于低秩适应(LoRA)的参数高效微调。性能通过基准指标与衡量与领域代码一致性的相似性指标进行评估。结果显示,提示类方法如少样本学习和RAG能在低成本下有效提升领域相关性,但对基准准确率提升有限;而基于LoRA的微调在多数任务中持续实现更高准确率与更强领域对齐。研究揭示了在灵活性、计算成本与性能之间权衡的实际路径。

原文摘要 · Abstract (English)

Large language models (LLMs) have demonstrated strong capabilities in generating executable code from natural language descriptions. However, general-purpose models often struggle in specialized programming contexts where domain-specific libraries, APIs, or conventions must be used. Customizing smaller open-source models offers a cost-effective alternative to relying on large proprietary systems. In this work, we investigate how smaller language models can be adapted for domain-specific code generation using synthetic datasets. We construct datasets of programming exercises across three domains within the Python ecosystem: general Python programming, Scikit-learn machine learning workflows, and OpenCV-based computer vision tasks. Using these datasets, we evaluate three customization strategies: few-shot prompting, retrieval-augmented generation (RAG), and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). Performance is evaluated using both benchmark-based metrics and similarity-based metrics that measure alignment with domain-specific code. Our results show that prompting-based approaches such as few-shot learning and RAG can improve domain relevance in a cost-effective manner, although their impact on benchmark accuracy is limited. In contrast, LoRA-based fine-tuning consistently achieves higher accuracy and stronger domain alignment across most tasks. These findings highlight practical trade-offs between flexibility, computational cost, and performance when adapting smaller language models for specialized programming tasks.

代码生成小模型LoRA领域定制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。