PROTEA让多智能体大模型工作流可调试、可迭代,提升准确率。
PROTEA: Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows

- 通过评分中间节点输出定位瓶颈,支持反向评估与提示修订。
- 文档检查准确率从64.3%提升至83.9%,推荐命中率从0.30升至0.38。
- 适合需要调试复杂多代理系统的开发者,可视化推理过程。
多智能体大模型工作流由多个角色专用的LLM调用组成,常优于单提示基线,但难以调试与优化。故障可能源于中间输出的细微错误并传递至下游节点,需开发者查看长序列日志并推断修改哪个代理。本文提出PROTEA,一个统一的离线、测试驱动的多智能体工作流改进接口。PROTEA执行工作流,使用可配置评分标准对中间节点输出打分,并在工作流图上叠加各节点状态与推理依据,以定位潜在瓶颈。针对最终答案有参考的复杂系统,PROTEA进行反向节点评估:从最终答案参考和图上下文生成候选节点预期,与实际输出对比。对选定节点,提供可编辑的前后提示修订,自动重跑并展示输出变化与得分轨迹。在两个接近生产的流程中,文档检查准确率从64.3%提升至83.9%,推荐Hit@5从0.30升至0.38。六名资深LLM开发者在初步研究中认可其图级定位、节点推理说明及可编辑提示修订功能。
原文摘要 · Abstract (English)
Multi-agent LLM workflows -- systems composed of multiple role-specific LLM calls -- often outperform single-prompt baselines, but they remain difficult to debug and refine. Failures can originate from subtle errors in intermediate outputs that propagate to downstream nodes, requiring developers to inspect long traces and infer which agent to modify. We present PROTEA, a unified interface for offline, test-driven improvement of multi-agent workflows. PROTEA executes a workflow, scores intermediate node outputs with configurable rubrics, and overlays per-node states and rationales on the workflow graph to localize likely bottlenecks. To support complex systems where final-answer references are the primary supervision, PROTEA performs backward node evaluation: it generates candidate node-level expectations from final-answer references and graph context, then compares them with observed node outputs. For selected nodes, PROTEA presents targeted prompt revisions as editable before/after comparisons, then automatically reruns and re-evaluates the workflow to show output changes and score trajectories within the same interface. In two production-adjacent workflows, PROTEA improved document-inspection accuracy from 64.3% to 83.9% and recommendation Hit@5 from 0.30 to 0.38. In a formative study with six experienced LLM developers, participants valued graph-level localization, per-node rationales, and editable before/after prompt revisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。