SAP Joule在代码生成任务中表现优异，位列全球前五。
Evaluating SAP Joule for Code Generation
- 使用HumanEval-X基准测试29个模型，评估Joule的代码生成能力。
- 在JavaScript代码生成上严格准确率达80.49%，排名第五。
- 首次系统评测SAP Joule，适合关注企业级AI助手的研究者。
SAP发布了其自研生成式模型SAP Joule，旨在支持多种生成任务，包括作为软件工程师的代码助手。尽管目前尚未聚焦于SAP特定的ABAP代码生成，但可应用于JavaScript等通用编程语言。本文通过HumanEval-X JavaScript基准，将SAP Joule与共计29个其他模型进行对比。结果显示，SAP Joule在严格准确率上达到80.49%，位列所有模型中的第五名。据我们所知，这是对SAP Joule代码生成能力的首次系统性比较评估。
原文摘要 · Abstract (English)
SAP has released its own proprietary generative model SAP Joule, intended for various generative tasks, including serving as a code assistant for software engineers. While Joule is yet not focused on SAP-specific ABAP code generation, it can be used for other common languages, including Javascript. This paper compares SAP Joules Javascript coding capabilities against a total of 29 other models using the HumanEval-X Javascript benchmark. SAP Joule achieves a strict accuracy of 80.49% as the fifth best model in our evaluation. To the best of our knowledge, this is the first comparative evaluation of SAP Joule code generation capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。