用动态题目和新指标评估大模型真实能力，发现它们常高估自己。
Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence
- 用可变参数的动态题库替代固定问答，防止记忆或猜测。
- 25个主流模型在复杂任务上表现差，简单题也常答错。
- 首次量化模型自信心与实际能力的差距，适合研究可靠性的人看。
随着机器智能发展，评估不同AI模型解决问题能力的需求日益增长。然而，现有基准测试通常过于简单，导致模型表现趋同，难以区分真实能力；且多采用静态问答对，易被模型记忆或猜中。为此，我们提出动态智能评估（DIA）方法，通过动态题目模板和改进指标，在数学、密码学、网络安全、计算机科学等多个领域评估模型。配套数据集DIA-Bench包含多样化的挑战模板，参数可变，形式涵盖文本、PDF、编译二进制文件、视觉谜题及CTF风格安全挑战。框架引入四项新指标，评估模型在多次尝试中的可靠性和置信度。结果表明，即使简单问题在不同形式下也频繁出错，暴露了模型可靠性严重不足。例如，GPT-4o API版本常高估其数学能力，而ChatGPT-4o因有效使用工具表现更优；自评估中，OpenAI o1-mini展现出最佳的任务判断力。我们评估了25个顶尖LLM，发现当前模型在复杂任务中普遍表现不佳，甚至对简单问题也常表现出意外低信心。DIA框架为评估问题解决能力、自适应智能及自我认知设定了新标准。数据集已公开于项目页：https://github.com/DIA-Bench。
原文摘要 · Abstract (English)
As machine intelligence evolves, the need to test and compare the problem-solving abilities of different AI models grows. However, current benchmarks are often simplistic, allowing models to perform uniformly well and making it difficult to distinguish their capabilities. Additionally, benchmarks typically rely on static question-answer pairs that the models might memorize or guess. To address these limitations, we introduce Dynamic Intelligence Assessment (DIA), a novel methodology for testing AI models using dynamic question templates and improved metrics across multiple disciplines such as mathematics, cryptography, cybersecurity, and computer science. The accompanying dataset, DIA-Bench, contains a diverse collection of challenge templates with mutable parameters presented in various formats, including text, PDFs, compiled binaries, visual puzzles, and CTF-style cybersecurity challenges. Our framework introduces four new metrics to assess a model's reliability and confidence across multiple attempts. These metrics revealed that even simple questions are frequently answered incorrectly when posed in varying forms, highlighting significant gaps in models' reliability. Notably, API models like GPT-4o often overestimated their mathematical capabilities, while ChatGPT-4o demonstrated better performance due to effective tool usage. In self-assessment, OpenAI's o1-mini proved to have the best judgement on what tasks it should attempt to solve. We evaluated 25 state-of-the-art LLMs using DIA-Bench, showing that current models struggle with complex tasks and often display unexpectedly low confidence, even with simpler questions. The DIA framework sets a new standard for assessing not only problem-solving but also a model's adaptive intelligence and ability to assess its limitations. The dataset is publicly available on the project's page: https://github.com/DIA-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。