测试发现主流模型在分布外任务中表现差，即使性能好也可能没学对组合特征。
Out-of-distribution generalisation is hard: evidence from ARC-like tasks
- 用两个带明确分布外指标的任务检验模型组合能力
- MLP/CNN/Transformer均无法在分布外任务中正确泛化
- 新架构虽能提升分布外性能，但仍可能学错关键特征
分布外（OOD）泛化被认为是人类与动物智能的标志。要通过组合实现OOD泛化，系统必须发现输入输出映射中的环境不变属性，并将其迁移至新输入。这要求系统能识别任务无关且可组合的输入特征及组合方式，从而基于特征组合而非数据点插值进行决策。我们提出，仅在分布外设置下测试不足以确认算法是否真正学习了组合结构，还需验证所识别特征是否具备组合性。为此，我们设计了两个具有明确分布外度量的任务，发现三种常用神经网络（MLP、CNN、Transformer）均无法解决。此外，我们提出了两种带有先验偏置的新网络架构，使其能在分布外场景中取得接近完美的性能。然而，即便如此，算法仍可能未能学习到正确的组合特征。
原文摘要 · Abstract (English)
Out-of-distribution (OOD) generalisation is considered a hallmark of human and animal intelligence. To achieve OOD through composition, a system must discover the environment-invariant properties of experienced input-output mappings and transfer them to novel inputs. This can be realised if an intelligent system can identify appropriate, task-invariant, and composable input features, as well as the composition methods, thus allowing it to act based not on the interpolation between learnt data points but on the task-invariant composition of those features. We propose that in order to confirm that an algorithm does indeed learn compositional structures from data, it is not enough to just test on an OOD setup, but one also needs to confirm that the features identified are indeed compositional. We showcase this by exploring two tasks with clearly defined OOD metrics that are not OOD solvable by three commonly used neural networks: a Multi-Layer Perceptron (MLP), a Convolutional Neural Network (CNN), and a Transformer. In addition, we develop two novel network architectures imbued with biases that allow them to be successful in OOD scenarios. We show that even with correct biases and almost perfect OOD performance, an algorithm can still fail to learn the correct features for compositional generalisation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。