用LASSO平台实现LLM代码生成可靠性的自动化测试
Test-driven Software Experimentation with LASSO: an LLM Prompt Benchmarking Example
- 构建可执行脚本化工具，支持快速设计和运行测试驱动实验
- 通过自包含脚本验证大模型代码生成的运行时可靠性
- 适合关注AI生成代码质量评估的研究者与开发者
实证软件工程面临一个关键挑战：缺乏标准化工具来快速开发和执行测试驱动的软件实验（TDSE），即通过执行软件主体并观察分析其实际运行时行为。本文提出通用分析平台LASSO，提供一组最小化的领域特定语言和数据结构，用于开展TDSE。LASSO通过可执行脚本语言，使用户能够高效评估运行时语义与执行特征，而不仅限于静态属性。我们展示了一个典型实例，利用自包含、可复用且可扩展的实验脚本，评估大语言模型在代码生成中的可靠性。LASSO平台及实时流水线示例已公开：https://softwareobservatorium.github.io/
原文摘要 · Abstract (English)
Empirical software engineering faces a critical gap: the lack of standardized tools for rapid development and execution of Test-Driven Software Experiments (TDSEs) -- that is, experiments that involve the execution of software subjects and the observation and analysis of their "de facto" run-time behavior. In this paper we present a general-purpose analysis platform called LASSO that provides a minimal set of domain-specific languages and data structures to conduct TDSEs. By empowering users with an executable scripting language to design and execute TDSEs, LASSO enables efficient evaluation of run-time semantics and execution characteristics in addition to statically determined properties. We present an example TDSE that demonstrates the practical benefits of LASSO's scripting capabilities for assessing the reliability of LLMs for code generation by means of a self-contained, reusable and extensible study script. The LASSO platform and live pipeline examples are publicly available at: https://softwareobservatorium.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。