构建高效可靠的AI评审系统，提升大模型软件的自动化评估质量。
Engineering AI Judge Systems
- 针对大模型软件动态性设计专用评审框架
- 评审准确率提升最高达6.2%，开发成本显著降低
- 适合大模型研发团队与质量保障工程师参考
AI评审系统旨在自动评估基于基础模型的软件（即FMware）。由于FMware固有的动态性和随机性，其评审系统的开发需独特的工程生命周期，面临新挑战。本文基于工业实践，分析了开发过程中存在的高耗时、高成本与判断不准等问题。提出一种新框架，旨在提升高质量AI评审系统开发效率。通过在提交信息生成型FMware上的案例研究验证，采用该框架开发的系统，其判断准确率相比传统方式最高提升6.2%，同时大幅减少开发投入。
原文摘要 · Abstract (English)
AI judge systems are designed to automatically evaluate Foundation Model-powered software (i.e., FMware). Due to the intrinsic dynamic and stochastic nature of FMware, the development of AI judge systems requires a unique engineering life cycle and presents new challenges. In this paper, we discuss the challenges based on our industrial experiences in developing AI judge systems for FMware. These challenges lead to substantial time consumption, cost and inaccurate judgments. We propose a framework that tackles the challenges with the goal of improving the productivity of developing high-quality AI judge systems. Finally, we evaluate our framework with a case study on judging a commit message generation FMware. The accuracy of the judgments made by the AI judge system developed with our framework outperforms those made by the AI judge system that is developed without our framework by up to 6.2%, with a significant reduction in development effort.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。