通过系统化提升逻辑不变题目的难度,更精准发现大模型的脆弱点。
LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
- 构建难度可量化的问题变体搜索框架,定位最易导致模型失效的改写形式。
- 难度上升时模型性能下降,推理错误显著增多,最高降幅达随机采样的5倍。
- 在高难度变体上微调,能带来更稳定的鲁棒性提升,适合模型安全评估与训练优化。
随着大语言模型(LLMs)越来越多地在无人监督下执行任务,其鲁棒性至关重要。理想情况下,当问题逻辑保持不变时,即使实体(如姓名、数字或上下文细节)发生变化,模型也不应失败。然而现有评估方法通常随机选取允许的变体,难以准确识别真正挑战模型的题目,易高估其鲁棒性。为此,我们提出逻辑不变难度增强(LPDS)框架,该框架(i)量化问题变体的难度,(ii)系统搜索所有合法变体以找到最难的样本,暴露模型缺陷。实验表明,随着难度增加,模型性能下降,推理链中的错误更加明显。此外,相比随机采样,LPDS能更高效地发现高难度变体,使性能下降幅度最大提升至5倍。最后,我们在更难的变体上进行微调,获得的鲁棒性提升比在简单变体上训练更一致有效。
原文摘要 · Abstract (English)
As large language models (LLMs) are increasingly deployed to perform tasks with minimal human oversight, it is crucial that these models operate robustly. In particular, a model that can solve a given problem should not fail simply because certain entities$\unicode{x2013}$such as names, numbers, or other contextual details$\unicode{x2013}$have changed while the underlying problem logic remains the same. Prior work suggests that current LLMs still struggle with this form of robustness: they often succeed on some variations of a problem but fail on others. However, existing evaluations often lack a systematic way to identify which logic-preserving variations are most likely to induce failure. Instead, they typically test a random subset of allowable variations, which can overstate robustness. To address this gap, we introduce logic-preserving difficulty scaling (LPDS), a framework that (i) quantifies the difficulty of a problem variation and (ii) systematically searches the space of allowable variations to find those that maximize difficulty and expose failures. We show that as difficulty increases, performance declines and errors in the models' reasoning chains become more pronounced. We further demonstrate that LPDS efficiently finds difficult problem variations for a model, resulting in performance drops up to 5 times larger compared to random sampling. Finally, we show that fine-tuning on more difficult variations leads to more consistent robustness gains than training on easier ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。