用低秩+傅里叶正则化，让代码模型跨语言生成更高效。
FLeX: Fourier-based Low-rank EXpansion for multilingual transfer
- 用低秩适配微调小数据集，只改少量参数。
- 傅里叶正则化使Java生成准确率提升至42.1%。
- 适合资源有限时快速适配多语言代码模型。
跨语言代码生成在多编程语言共存的企业环境中至关重要。然而，为每种语言单独微调大模型计算开销巨大。本文研究参数高效微调方法与优化器改进对从Python向Java等语言迁移的效果。我们使用低秩适配（LoRA）微调Code Llama 7B模型，在小而高质量的数据集MBPP上仅优化少量参数，并对比Adam与Sophia优化器，同时探索一种新的傅里叶正则化技术。结果表明：(1) 在小数据集上使用LoRA微调的模型在pass@1指标上达到40.1%，优于广泛微调的Code Llama-Python-7B模型的38.4%；(2) Sophia收敛更快，但最终性能差异微弱；(3) 傅里叶正则化显著提升跨语言迁移能力，使Java任务的pass@1达到42.1%，远超34.2%的基线。结果表明，结合LoRA、优化训练策略与频域正则化，可高效实现单语言大模型在多语言代码生成中的良好表现。
原文摘要 · Abstract (English)
Cross-lingual code generation is critical in enterprise environments where multiple programming languages coexist. However, fine-tuning large language models (LLMs) individually for each language is computationally prohibitive. This paper investigates whether parameter-efficient fine-tuning methods and optimizer enhancements can improve cross-lingual transfer from Python to languages like Java. We fine-tune the Code Llama 7B model using low-rank adaptation (LoRA) to optimize a small subset of parameters and compare Adam and Sophia optimizers, while exploring a novel Fourier-based regularization technique. Our contributions include: (1)demonstrating that LoRA fine-tuning on a small, high-quality dataset (MBPP) can exceed the pass@1 performance of the more broadly fine-tuned Code Llama-Python-7B model (40.1% vs. 38.4%); (2) showing that while Sophia achieves faster convergence than Adam, final pass@1 scores show marginal differences; and (3) presenting evidence that Fourier-based regularization during fine-tuning significantly improves cross-lingual transfer, achieving 42.1% pass@1 on Java tasks compared to the 34.2% baseline. These findings suggest that combining LoRA, optimized training methods, and frequency-domain regularization can efficiently adapt single-language LLMs to perform well across multiple programming languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。