让测试用例懂业务,自动生成能发现数据错误的智能测试。
PrismaDV: Automated Task-Aware Data Unit Test Generation

- 分析下游代码和数据特征,生成与任务相关的测试用例。
- 在60个任务上表现优于现有方法,有效捕捉数据错误影响。
- 用反馈优化提示词,自动提升测试生成效果,适合数据工程师。
数据是现代企业核心资源,数据验证对保障下游应用可靠性至关重要。然而现有自动化数据单元测试框架大多任务无关:仅验证数据,不考虑消费数据的代码语义与需求。我们提出PrismaDV,一个复合式AI系统,通过分析下游任务代码与数据集特征,识别数据访问模式,推断隐含数据假设,并生成任务感知的可执行数据单元测试。为持续适应特定数据集与任务,我们提出SIFTA(选择性信息反馈任务自适应)框架,利用测试与任务执行结果稀缺信号优化提示词。我们在两个新基准上评估,涵盖5个数据集上的60项任务,结果显示PrismaDV在生成反映数据错误端到端影响的测试用例方面,始终优于任务无关与任务相关基线。此外,SIFTA可自动学习优于人工编写或通用提示优化器生成的提示词。我们已公开发布基准与原型实现。
原文摘要 · Abstract (English)
Data is a central resource for modern enterprises, and data validation is essential for ensuring the reliability of downstream applications. However, existing automated data unit testing frameworks are largely task-agnostic: they validate datasets without considering the semantics and requirements of the code that consumes the data. We present PrismaDV, a compound AI system that analyzes downstream task code together with dataset profiles to identify data access patterns, infer implicit data assumptions, and generate task-aware executable data unit tests. To further adapt the data unit tests over time to specific datasets and downstream tasks, we propose "Selective Informative Feedback for Task Adaptation" (SIFTA), a prompt-optimization framework that leverages the scarce outcomes from the execution of data unit tests and downstream tasks. We evaluate PrismaDV on two new benchmarks spanning 60 tasks across five datasets, where it consistently outperforms both task-agnostic and task-aware baselines in generating unit tests that reflect the end-to-end impact of data errors. Furthermore, we show that with SIFTA, we can automatically learn prompts for PrismaDV's modules that outperform prompts written by hand or generated from a generic prompt optimizer. We publicly release our benchmarks and prototype implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。