用动态生成问题的智能体框架,更真实地评估大模型推理能力。
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
- 构建教师-协调器-学生三智体动态生成与验证问题。
- 自动提升难度并暴露传统评测忽略的逻辑错误。
- 适合研究大模型进化和评测体系的学者参考。
大语言模型的评估长期依赖静态数据集,难以反映模型推理能力的演进。为此,我们提出一种以智能体为中心的动态评测范式:自主智能体迭代生成、验证并求解问题。其中,教师智能体生成候选问题,协调器智能体严格验证有效性并防范对抗攻击,学生智能体尝试求解。若问题无效,教师智能体修正直至通过;若学生正确求解,协调器则要求教师生成更难变体。该机制随智能体能力提升自动升级难度,实现无需人工标注数据的渐进式评估。采用文本异常检测作为核心评估形式,需跨句逻辑推理且规避模式匹配捷径,我们验证该协议能系统揭示传统基准未发现的边缘案例推理错误。进一步建议从跨模型对比、初始与最终问题间进展等多维度评估。本方法将评估重心从固定数据集转向动态协议,为持续演进的语言模型提供可持续评估路径,并推动智能体驱动评测体系的协同演化研究。
原文摘要 · Abstract (English)
The evaluation of large language models (LLMs) has predominantly relied on static datasets, which offer limited scalability and fail to capture the evolving reasoning capabilities of recent models. To overcome these limitations, we propose an agent-centric benchmarking paradigm that moves beyond static datasets by introducing a dynamic protocol in which autonomous agents iteratively generate, validate, and solve problems. Within this protocol, a teacher agent generates candidate problems, an orchestrator agent rigorously verifies their validity and guards against adversarial attacks, and a student agent attempts to solve the validated problems. An invalid problem is revised by the teacher agent until it passes validation. If the student correctly solves the problem, the orchestrator prompts the teacher to generate more challenging variants. Consequently, the benchmark scales in difficulty automatically as more capable agents are substituted into any role, enabling progressive evaluation of large language models without manually curated datasets. Adopting text anomaly detection as our primary evaluation format, which demands cross-sentence logical inference and resists pattern-matching shortcuts, we demonstrate that this protocol systematically exposes corner-case reasoning errors that conventional benchmarks fail to reveal. We further advocate evaluating systems along several complementary axes including cross-model pairwise performance and progress between the initial and orchestrator-finalized problems. By shifting the focus from fixed datasets to dynamic protocols, our approach offers a sustainable direction for evaluating ever-evolving language models and introduces a research agenda centered on the co-evolution of agent-centric benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。