用大模型自动检测代码质量问题，提升大型团队协作效率。
A Note on Code Quality Score: LLMs for Maintainable Large Codebases
- 用微调后的Llama3识别编码规范问题和生成代码审查意见。
- 系统在真实场景中用户帮助度达每周60%以上，精度表现优秀。
- 适合大规模团队开发者、代码审查工具优化者使用。
大型软件系统中，多工程师协同开发给代码质量维护带来巨大挑战。本文提出代码质量评分系统（CQS），通过两个经过SFT与离线强化学习微调的Llama3模型，自动检测代码变更中的常见质量问题，并为大模型生成的代码审查提供有效反馈。为保障用户体验，系统叠加人工规则过滤错误输出与幻觉。离线评估显示，该系统在识别有效问题方面具有优异精确率。目前该系统已在工业级规模部署，持续实现每周60%的用户帮助度，证明其在真实环境中的有效性。本文还分享了基于开发者反馈构建训练数据的经验。
原文摘要 · Abstract (English)
Maintaining code quality in large-scale software systems presents significant challenges, particularly in settings where a large numbers of engineers work concurrently on a codebase. This paper introduces Code Quality Score (CQS) system to automatically detect issues with a set of code changes and provide actionable insights. At its core, the CQS system is powered by two Llama3 models, fine-tuned (with SFT and offline RL approaches), to a) detect common code quality issues related to coding best practices and b) to provide good ``critiques'' for LLM-generated code review respectively. To maintain good user experience, we layer the system with hand-crafted rules to filter out incorrect responses/hallucinations. Offline evaluations show that our CQS system is able to achieve an impressive precision rate for identifying valid issues. This system has already been rolled out to developers in an industrial scale setting and has consistently achieved 60\% week over week user helpfulness rate, demonstrating its effectiveness in a real-world environment. In this paper, we present details of the CQS system along with some learnings on curating developer feedback to create training data for LLM fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。