用动态问答评估大模型，更精准发现知识短板
JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
- 构建可自主生成问题的智能代理，系统遍历知识图谱
- 通过多轮对话自适应调整难度，避免数据污染
- 适合需要深度优化知识能力的模型研发团队
当前大模型评估主要依赖静态基准，存在知识覆盖有限和难度固定的问题，导致评估流于表面，难以支撑模型针对性优化。为此，我们提出JudgeAgent，一种基于知识驱动的动态评估框架。为解决知识覆盖不足，该框架利用配备上下文图的LLM智能体，系统化遍历知识结构生成问题；为缓解数据污染和难度不匹配，采用难度自适应的多轮访谈机制。实验表明，JudgeAgent能实现更全面的评估，推动模型有效迭代，展现出知识驱动动态评估的潜力。代码已开源。
原文摘要 · Abstract (English)
Current evaluation methods for large language models (LLMs) primarily rely on static benchmarks, presenting two major challenges: limited knowledge coverage and fixed difficulties that mismatch with the evaluated LLMs. These limitations lead to superficial assessments of LLM knowledge, thereby impeding the targeted model optimizations. To bridge this gap, we propose JudgeAgent, a knowledge-driven and dynamic evaluation framework for LLMs. To address the challenge of limited knowledge coverage, JudgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures systematically for question generation. Furthermore, to mitigate data contamination and difficulty mismatch, it adopts a difficulty-adaptive and multi-turn interview mechanism. Thereby, JudgeAgent can achieve comprehensive evaluations and facilitate more effective improvement of LLMs. Empirical results demonstrate that JudgeAgent enables more comprehensive evaluations and facilitates effective model iterations, highlighting the potential of this knowledge-driven and dynamic evaluation paradigm. The source code is available on https://github.com/DataArcTech/JudgeAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。