arXiv:2607.05638cs.SEcs.AI2026-07

用分维评估诊断模型缺陷,让AI系统迭代更精准有效

EvalLoop: A Methodology for Evaluation-Driven Iterative Improvement of Business AI Systems

论文配图:EvalLoop: A Methodology for Evaluation-Driven Iterative Improvement of Business AI Systems
图 1 · 摘自论文原文
  • 将评估拆解为维度分组、故障分类和结构化迭代三机制
  • 诊断出69%幻觉由提示词引发,优化后准确率提升16.8个百分点
  • 适合需要持续改进的业务级大模型团队使用

在业务场景中部署大语言模型时,多数评估体系仅静态比对模型性能,忽视了诊断系统失效原因并指导改进的核心价值。本文提出EvalLoop方法论,通过三项机制实现评估驱动的迭代优化:(1)维度指标分组,将质量分解为业务相关维度,支持独立故障诊断;(2)故障模式分类,识别弱维度中的失败原因,连接诊断与行动;(3)结构化迭代流程,每次只变更一个系统变量,对比前后维度表现。案例研究涵盖10个模型、3家供应商、18项指标、5个维度、3轮迭代。维度诊断发现69%的幻觉源于提示词引发的误解,不可见于整体评分。针对性优化提示词后,最佳模型整体表现从82.6%提升至94.6%,内容准确性提升16.8个百分点,合成能力提升26.4个百分点。此前无诊断的配置调整未带来任何改善,凸显诊断缺失的成本。此外,维度分析可支持定制化部署选型;一次盲评(4模型,16案例)即确认维度排名,相较全量评估减少94%评审负担。所有流程均已封装为可复用工具包。

原文摘要 · Abstract (English)

Teams deploying large language models in business contexts need evaluation systems, yet most treat evaluation as static model selection: run benchmarks, rank models, deploy the winner. This framing misses evaluation's primary value for production systems--diagnosing why a system underperforms and guiding what to fix. We present EvalLoop, a methodology for evaluation-driven iterative improvement. EvalLoop organizes evaluation around three mechanisms: (1) dimensional metric grouping that decomposes quality into business-relevant dimensions enabling orthogonal failure diagnosis; (2) failure mode classification that categorizes why outputs fail within weak dimensions, bridging diagnosis to action; and (3) a structured iteration workflow where each evaluation run varies one system variable and compares dimensional profiles before and after. We validate EvalLoop through a case study on sales intelligence briefing generation (10 models, 3 providers, 18 metrics, 5 dimensions, 3 iterations). Dimensional diagnosis identified that 69% of hallucination failures were prompt-induced interpretation errors--invisible in aggregate scoring. A targeted prompt fix improved the best model from 82.6% to 94.6% overall, with improvement concentrated in diagnosed dimensions (Content Accuracy +16.8pp, Synthesis Power +26.4pp). An undirected configuration change in a prior iteration produced zero impact, illustrating the cost of iterating without diagnosis. We additionally demonstrate that dimensional profiling enables deployment-specific model selection, and that a one-time blind human gate on a finalist panel (4 models, 16 cases) confirms dimensional rankings while resolving multi-criteria deployment trade-offs--a 94% reduction in review burden compared to evaluating the full design. EvalLoop is packaged as reusable artifacts (playbook, agent specification, template repository) for adoption by other teams.

评估方法大模型迭代优化业务落地

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。