新基准NoRA挑战神经模型突破路径依赖的推理局限
When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning
- 构建多层复杂场景，要求模型超越简单路径组合
- 现有方法在旧基准表现好，但在新基准上大幅退化
- 适合研究可泛化神经推理与认知建模的研究者
设计能系统性推理的模型是长期存在的挑战。近年来，针对系统性关系推理提出了多种方案，包括神经符号方法、Transformer变体和专用图神经网络。然而，现有基准过于简化，假设推理可归结为关系路径的组合。这一假设被硬编码进多个最新模型中，导致其在旧基准表现优异但难以泛化。为推动神经网络系统性关系推理的发展，我们提出NoRA新基准，引入多层级难度，要求模型突破路径依赖式推理。
原文摘要 · Abstract (English)
Designing models that can learn to reason in a systematic way is an important and long-standing challenge. In recent years, a wide range of solutions have been proposed for the specific case of systematic relational reasoning, including Neuro-Symbolic approaches, variants of the Transformer architecture, and specialised Graph Neural Networks. However, existing benchmarks for systematic relational reasoning focus on an overly simplified setting, based on the assumption that reasoning can be reduced to composing relational paths. In fact, this assumption is hard-baked into the architecture of several recent models, leading to approaches that can perform well on existing benchmarks but are difficult to generalise to other settings. To support further progress in the field of systematic relational reasoning with neural networks, we introduce NoRA, a new benchmark which adds several levels of difficulty and requires models to go beyond path-based reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。