构建LLM/RAG应用的上线评估框架，融合可观测性与持续集成
LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications
- 通过自动化测试、可观测性与质量门禁形成可部署工作流
- 在162个实验组合中验证，不同模型在延迟与准确率间存在权衡
- 适合需要安全上线LLM系统的工程团队使用
我们提出一种用于LLM和RAG应用的就绪性检查工具，将评估转化为部署决策流程。系统在最小API契约下整合自动化基准测试、OpenTelemetry可观测性与CI质量门禁，聚合工作流成功率、策略合规性、事实一致性、检索命中率、成本及p95延迟，生成场景加权的就绪评分，并绘制帕累托前沿。在票务路由任务和BEIR事实核验任务（SciFact与FiQA）上进行全面的Azure矩阵验证（162/162有效组合，涵盖数据集、场景、检索深度、种子与模型）。结果表明就绪性并非单一指标：在FiQA任务中，k=5且以SLA优先时，gpt-4.1-mini在就绪度与忠实度上领先，而gpt-5.2付出显著延迟代价；在SciFact上模型质量相近但仍可操作区分。票务路由回归门禁持续拒绝不安全提示变体，证明该框架能阻止风险发布，而非仅报告离线分数。最终形成可复现、面向实际运维的决策框架。
原文摘要 · Abstract (English)
We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow. The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract, then aggregates workflow success, policy compliance, groundedness, retrieval hit rate, cost, and p95 latency into scenario-weighted readiness scores with Pareto frontiers. We evaluate the harness on ticket-routing workflows and BEIR grounding tasks (SciFact and FiQA) with full Azure matrix coverage (162/162 valid cells across datasets, scenarios, retrieval depths, seeds, and models). Results show that readiness is not a single metric: on FiQA under sla-first at k=5, gpt-4.1-mini leads in readiness and faithfulness, while gpt-5.2 pays a substantial latency cost; on SciFact, models are closer in quality but still separable operationally. Ticket-routing regression gates consistently reject unsafe prompt variants, demonstrating that the harness can block risky releases instead of merely reporting offline scores. The result is a reproducible, operationally grounded framework for deciding whether an LLM or RAG system is ready to ship.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。