测试大模型在视觉模糊下的类比推理能力，发现性能大幅下降。
Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?
- 用噪声和属性平滑模拟感知不确定性，测试模型鲁棒性。
- o3-mini准确率从86.6%降至17.0%，DeepSeek R1从80.6%降至23.2%。
- 神经符号模型ARLC表现稳定，适合研究抗干扰推理方法者参考。
本文首次评估了两种先进大推理模型（OpenAI o3-mini 和 DeepSeek R1）在类比推理任务中的表现，基于经典的非语言人类智商测试Raven's Progressive Matrices。使用I-RAVEN及其扩展版本I-RAVEN-X进行基准测试，后者考察长推理规则与属性值范围的泛化能力。为评估视觉不确定性的影响，我们扩展了I-RAVEN-X数据集，引入随机干扰属性和输入属性值平滑处理以模拟不完美感知。结果显示，o3-mini在原始I-RAVEN上准确率为86.6%，但在更复杂的I-RAVEN-X上骤降至17.0%（接近随机水平），且推理消耗增加3.4倍；DeepSeek R1也从80.6%降至23.2%。相比之下，神经符号概率归纳模型ARLC在所有分布外测试中保持稳健，准确率仅从98.6%微降至88.0%。代码已开源。
原文摘要 · Abstract (English)
This work presents a first evaluation of two state-of-the-art Large Reasoning Models (LRMs), OpenAI's o3-mini and DeepSeek R1, on analogical reasoning, focusing on well-established nonverbal human IQ tests based on Raven's progressive matrices. We benchmark with the I-RAVEN dataset and its extension, I-RAVEN-X, which tests the ability to generalize to longer reasoning rules and ranges of the attribute values. To assess the influence of visual uncertainties on these symbolic analogical reasoning tests, we extend the I-RAVEN-X dataset, which otherwise assumes an oracle perception. We adopt a two-fold strategy to simulate this imperfect visual perception: 1) we introduce confounding attributes which, being sampled at random, do not contribute to the prediction of the correct answer of the puzzles, and 2) we smoothen the distributions of the input attributes' values. We observe a sharp decline in OpenAI's o3-mini task accuracy, dropping from 86.6% on the original I-RAVEN to just 17.0% -- approaching random chance -- on the more challenging I-RAVEN-X, which increases input length and range and emulates perceptual uncertainty. This drop occurred despite spending 3.4x more reasoning tokens. A similar trend is also observed for DeepSeek R1: from 80.6% to 23.2%. On the other hand, a neuro-symbolic probabilistic abductive model, ARLC, that achieves state-of-the-art performances on I-RAVEN, can robustly reason under all these out-of-distribution tests, maintaining strong accuracy with only a modest accuracy reduction from 98.6% to 88.0%. Our code is available at https://github.com/IBM/raven-large-language-models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。