测试大模型在逻辑混淆下的表现，发现其推理能力严重下降。
Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation
- 设计结构保持的逻辑混淆框架Logifus，生成四类推理任务数据。
- 六款主流模型在混淆题上平均性能下降22%至47%，最高达47%。
- 适合关注模型深层理解能力、鲁棒性评估的研究者参考。
标准形式下，大语言模型（LLMs）能较好完成算术求解、真值表判断和三段论等任务，但在逻辑等价但形式混淆的问题上常出现失败。为此，我们提出Logifus——一种结构保持的逻辑混淆框架，并基于此构建首个诊断性基准LogiQAte，包含1,108个问题，涵盖四类推理任务：(i) Obfus FOL（保等价重写的一阶逻辑蕴含），(ii) Obfus Blood Relation（间接亲属关系图蕴含），(iii) Obfus Number Series（符号替换下的数列模式归纳），(iv) Obfus Direction Sense（方向与参考系改变下的导航推理）。在所有任务中，对六种顶尖模型进行评估，发现混淆导致零样本性能显著下降：GPT-4o平均下降47%，GPT-5下降27%，o4-mini下降22%。结果表明当前模型依赖表面形式而非深层理解，亟需构建真正具备语义保持能力的模型。
原文摘要 · Abstract (English)
Tasks such as solving arithmetic equations, evaluating truth tables, and completing syllogisms are handled well by large language models (LLMs) in their standard form, but they often fail when the same problems are posed in logically equivalent yet obfuscated formats. To study this vulnerability, we introduce Logifus, a structure-preserving logical obfuscation framework, and, utilizing this, we present LogiQAte, a first-of-its-kind diagnostic benchmark with 1,108 questions across four reasoning tasks: (i) Obfus FOL (first-order logic entailment under equivalence-preserving rewrites), (ii) Obfus Blood Relation (family-graph entailment under indirect relational chains), (iii) Obfus Number Series (pattern induction under symbolic substitutions), and (iv) Obfus Direction Sense (navigation reasoning under altered directions and reference frames). Across all the tasks, evaluating six state-of-the-art models, we find that obfuscation severely degrades zero-shot performance, with performance dropping on average by 47% for GPT-4o, 27% for GPT-5, and 22% for reasoning model, o4-mini. Our findings reveal that current LLMs parse questions without deep understanding, highlighting the urgency of building models that genuinely comprehend and preserve meaning beyond surface form.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。