通过多响应联合评估,让裁判智能体更全面地发现推理问题并提升主代理性能。
JAF: Judge Agent Forest
- 裁判智能体同时评估多个主代理生成的问答对,捕捉跨实例模式与矛盾。
- 在云配置误报排查任务中,使主代理错误率降低37.6%,显著优于单点评估。
- 适用于需要高可靠推理的复杂决策系统,如安全检测与自动化运维。
裁判智能体是智能体框架的核心:提供自动评估能力,并支持推理过程的迭代自我优化。本文提出JAF(Judge Agent Forest)框架,其中裁判智能体对主智能体生成的一组查询-响应对进行联合推理,而非逐个孤立评估。这一范式将裁判从局部评价者转变为整体学习者:通过同步评估相关响应,裁判能识别跨实例的模式与不一致,其聚合反馈使主智能体得以从裁判的集体视角审视自身输出。概念上,JAF融合信念传播与集成学习原理:重叠的上下文邻域形成知识图结构,促进批判信息传播;重复随机评估生成上下文敏感的稳健判断集合。JAF可通过提示学习(ICL)完全实现,裁判针对每个查询,使用对应主代理响应及少量可能带噪的同侪样例进行提示。尽管嵌入空间中的kNN是样例的自然起点,但该方法忽略了类别结构、领域元数据或现代大模型可识别的细微差别。为克服这些局限,我们设计了一种灵活的局部敏感哈希(LSH)算法,通过整合语义嵌入、大模型驱动的哈希谓词、类别标签监督及相关附加信息,学习具有信息量的二进制编码。这些编码支持高效、可解释且关系感知的多样化样例选择,并进一步优化了思维链(CoT)推理路径的探索。我们在大规模云环境中的云配置误报排查这一高难度任务上验证了JAF的有效性。
原文摘要 · Abstract (English)
Judge agents are fundamental to agentic AI frameworks: they provide automated evaluation, and enable iterative self-refinement of reasoning processes. We introduce JAF: Judge Agent Forest, a framework in which the judge agent conducts joint inference across a cohort of query--response pairs generated by a primary agent, rather than evaluating each in isolation. This paradigm elevates the judge from a local evaluator to a holistic learner: by simultaneously assessing related responses, the judge discerns cross-instance patterns and inconsistencies, whose aggregate feedback enables the primary agent to improve by viewing its own outputs through the judge's collective perspective. Conceptually, JAF bridges belief propagation and ensemble-learning principles: overlapping in-context neighborhoods induce a knowledge-graph structure that facilitates propagation of critique, and repeated, randomized evaluations yield a robust ensemble of context-sensitive judgments. JAF can be instantiated entirely via ICL, with the judge prompted for each query using its associated primary-agent response plus a small, possibly noisy set of peer exemplars. While kNN in embedding space is a natural starting point for exemplars, this approach overlooks categorical structure, domain metadata, or nuanced distinctions accessible to modern LLMs. To overcome these limitations, we develop a flexible locality-sensitive hashing (LSH) algorithm that learns informative binary codes by integrating semantic embeddings, LLM-driven hash predicates, supervision from categorical labels, and relevant side information. These hash codes support efficient, interpretable, and relation-aware selection of diverse exemplars, and further optimize exploration of CoT reasoning paths. We validate JAF with an empirical study on the demanding task of cloud misconfigs triage in large-scale cloud environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。