让大模型生成后由测试严格验证,提升生产级输出可靠性。
TAG: A Lightweight Framework for Test-Driven Agentic Artifact Generation

- 生成时嵌入测试反馈,让模型根据错误信息自我修正。
- 结合确定性测试与大模型评估,覆盖语法到语义的多层验证。
- 用专家判断训练评估模型,实现可复用的专业级质量控制。
使用大语言模型生成结构化成果(如数据库查询、威胁框架映射、实体模式)相对简单,但要达到生产部署所需的可靠性却面临挑战。本文提出TAG轻量级框架,核心理念为‘大模型生成,我们验证’。该框架包含三大特性:首先,测试驱动生成——当测试失败时,大模型接收具体错误提示,理解失败原因并优化后续输出;其次,确定性与基于大模型的测试相结合——前者检测可编程验证的特征(如模式、语法、交叉引用),后者评估难以程序化检验的语义特征(如意图对齐、逻辑一致性、领域正确性);第三,专家提炼的评判器——基于大模型的测试通过模仿人类专家的判断分布,将人工质量审查转化为可扩展、可复用的评估代理,体现专业级标准。我们在微软Sentinel中对三种安全领域成果(KQL查询生成、MITRE ATT&CK映射、实体映射)进行了部署验证,结果表明该框架可广泛应用于其他生成任务,实现高质量输出的同时保持大模型生成的效率优势。
原文摘要 · Abstract (English)
Generating structured artifacts with Large Language Models - e.g.\ database queries, threat framework mappings, entity schemas - is relatively straightforward; however, making them reliable enough for production deployments presents challenges. We present TAG, a lightweight framework based on a core principle: \textit{LLMs generate, we validate}. This reframing shifts responsibility from generation quality to validation rigor. The framework rests on three key attributes: First, \textbf{test driven generation}: when tests fail, the LLM receives indicative error messages that expose why the output failed, enabling the LLM to understand its mistakes and refine subsequent attempts. Second, \textbf{deterministic and LLM-based tests}: deterministic tests catch heuristics that can be programmatically verified (schema, syntax, cross-reference), while LLM-based tests evaluate nuanced semantic and delicate features that resist programmatic inspection (intent alignment, logical consistency, domain correctness). Third, \textbf{expert-distilled judges}: LLM-based tests are calibrated to distill and replicate human expert decision distribution, transforming manual human quality gates into scalable, reusable evaluation proxies that reflect professional-grade validation standards. We demonstrate the framework on three artifact types in the security domain - KQL query generation, MITRE ATT\&CK mapping, and entity mapping - deployed in production at Microsoft Sentinel. We believe this framework can be applied beyond security to other artifact generation tasks, providing a path to reliable, high-quality outputs without sacrificing the efficiency gains of LLM generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。