让高阶需求与底层测试自动对齐,生成可审计的验证报告。
Integrating High-Level Requirements to Low-Level Tests with Machine-Readable V&V Specifications

- 将用户需求转为机器可读规范,自动分解并关联测试用例
- 36个需求、449次测试在持续集成中完成,线性扩展支持万级需求
- 适合需要合规审计的AI系统和嵌入式开发团队
现代软件团队拥有成熟的低层测试工具(如pytest、JUnit、Jest),可低成本编写单元测试并每次提交即运行。与此同时,系统工程发展出严谨的设计验证与确认(V&V)原则,有效将用户期望与交付成果对齐。然而实践中两者极少联动,高阶需求与实际执行的底层测试之间的联系往往靠人工维护,甚至缺失。这一断层对具备AI功能和网络物理特性的系统日益造成成本压力——监管机构要求提供可追溯的证据,证明高阶需求已满足,而原始测试结果缺乏所需的结构化信息。本文提出VNVSpec,一个开源框架,使V&V规范具备机器可读性和可执行性。用户可直接声明高阶需求,或从基于公开标准的目录导入。框架会检查需求质量,将其分解为含明确指标与验收标准的模块级需求,通过可追溯图将需求与测试结果关联,并整合证据生成判定结论与审计可用报告。我们通过自评估方式验证该框架:在持续集成中对其自身36个需求、449项测试进行周期性评估,耗时有限且呈线性增长,理论上可处理最多10,000个需求。此外,还探讨了该框架如何扩展用于黑盒AI模型及AI编程代理的测试。框架代码、完整测试套件、需求目录与基准脚本均开源,地址见 https://github.com/ai-vnv/vnvspec。
原文摘要 · Abstract (English)
Modern software teams have mature tools for low-level testing, such as pytest, JUnit, and Jest, which make it inexpensive to write unit tests and run them on every commit. Systems engineering, in parallel, has developed rigorous principles for design verification and validation (V&V), which has worked very well across engineering discipline to align user expecations and requirements with developers' deliverables. In practice, however, the two rarely connect, and the link between users' high-level requirements and the low-level tests that machines actually run is maintained by hand, if at all. This gap is increasingly costly for AI-enabled and cyber-physical systems, for which regulators now ask for traceable evidence that high-level requirements are met, while raw test results provide little of the structure such evidence requires. We introduce VNVSpec, an open-source framework that makes V&V specifications machine-readable and executable. With this framework, users state high-level requirements directly or import them from catalogs derived from published standards. Then, the framework checks requirement quality, supports decomposition into module-level requirements with explicit metrics and acceptance criteria, links these requirements to test results through a traceability graph, and compiles the collected evidence into verdicts and audit-ready reports. We evaluate the framework by self-application, in which it is continuously assessed in CI against its own specification of 36 requirements verified by 449 tests, completed within limited time which scales linearly and thus can handle up to 10,000 requirements. We also discuss how the framework extends to testing black-box AI models and AI coding agents. The framework, its full test suite, the catalogs, and the benchmark scripts are available at https://github.com/ai-vnv/vnvspec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。