用测试驱动开发让小模型在孟加拉语代码生成上达到98%准确率
Enhancing LLM Code Generation Capabilities through Test-Driven Development and Code Interpreter
- 结合测试驱动开发与代码解释器，无需微调即可提升代码生成
- 使用孟加拉语提示时整体准确率达85%，最小模型达98%精度
- 适合资源受限地区开发者，开源可复现，支持本地语言编程
近年来，提升大语言模型代码生成能力是自然语言处理的重要方向。尽管孟加拉语拥有2.42亿母语使用者，但训练大模型时却很少被关注。近期虽有多种微调和增强生成技术显著提升性能，但对用户而言仍需大量专业知识与资源。本文旨在降低资源匮乏新兴市场中强大代码生成工具的使用门槛，使用户能以母语进行编程。我们提出一种结合测试驱动开发（TDD）与代码解释器（CI）的新方法，采用开源权重模型，在孟加拉语提示下实现基线准确率提升，整体准确率达85%。该方法无需微调，甚至最小模型相比最大模型也能达到最高98%准确率。所有结果均公开于GitHub，可供验证与复现。
原文摘要 · Abstract (English)
Over the past few years, improving LLM code generation capabilities has been a key focus in NLP research. Despite Bengali having 242 million native speakers worldwide, it receives little attention when it comes to training LLMs. More recently, various fine-tuning and augmented generation techniques have been employed to significantly enhance code generation performance. However, they require considerable expertise and resources to utilize effectively as an end user. The goal of our work is to democratize access to powerful code generation tools in resource-constrained emerging markets, enabling users to leverage them in their native language. We introduce a novel approach that combines Test-Driven Development (TDD) and Code Interpreter (CI), utilizing open-weight models, which improves the baseline accuracy for code generation with Bengali prompts and achieves an overall accuracy of 85%. Our approach requires no finetuning and proves that even the smallest models in the same family can attain up to 98% accuracy compared to the largest models. All of our results are publicly shared in GitHub for validation and reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。