arXiv:2605.06213cs.AI2026-05

动态定位模型能力边界,实现更精准的LLM评估

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

  • 通过主动搜索模型在0.5成功率处的边界点进行评估
  • 覆盖安全、能力、真实性三类共4个维度的校准题库
  • 仅需API访问即可适配新模型,适合模型开发者与评测人员

当前大语言模型评估依赖固定基准测试,所有模型使用相同题目,导致天花板与地板效应掩盖真实能力差异。本文提出动态边界评估(DBE),聚焦于随机解码下每题通过概率接近0.5的边界区域,主动定位各模型的边界并将其置于全局可比难度量表上。DBE生成三类成果:(i) 经9个参考大模型验证的校准题库,涵盖安全、能力与真实性,含每题难度标签;(ii) 技能引导边界搜索(SGBS)算法,仅需API调用即可为目标模型寻找边界题;(iii) 评估协议,可将新模型映射到统一能力尺度,并在目标超出题库范围时自适应扩展。本方法应用于四大类别:安全(有害请求拒绝与过度拒绝)、能力(受限指令遵循)、真实性(多轮奉承抵抗)。评估覆盖更广模型谱系且无饱和现象,同时兼容现有数据集。

原文摘要 · Abstract (English)

Evaluating large language models (LLMs) today rests on fixed benchmarks that apply the same set of items to any model, producing ceiling and floor effects that mask capability gaps. We argue that the most informative evaluation signal lies at the boundary, where the per-prompt pass probability is near $0.5$ under random-sampling decoding, and propose Dynamic Boundary Evaluation (DBE), which actively locates each model's boundary and places it on a globally comparable difficulty scale. DBE delivers three artifacts: (i) a calibrated item bank covering safety, capability, and truthfulness, with per-item difficulty labels validated across $9$ reference LLMs; (ii) Skill-Guided Boundary Search (SGBS), a search algorithm that finds boundary items for a given target LLM using only API-level query access; and (iii) an evaluation protocol that places a new LLM on a unified ability scale and grows the evaluation set adaptively when the target falls outside the bank's coverage. We instantiate DBE on four categories spanning safety (harmful request refusal and over-refusal), capability (constrained instruction following), and truthfulness (multi-turn sycophancy resistance). The resulting evaluation covers a broader model spectrum without saturation while remaining compatible with existing datasets.

模型评估边界检测大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。