Transformer 能学会网格图的连通性，但对复杂图效果差。
Transformers Can Learn Connectivity in Some Graphs but Not Others
- 用不同规模的 Transformer 在生成的有向图上训练，测试其推理连通性的能力。
- 在网格图上表现良好，维度越高越难；非网格图且组件多时几乎无法学习。
- 模型越大，对网格图的泛化能力越强，但对复杂结构无改善。
推理能力对确保基于 Transformer 的大语言模型（LLM）回答的事实正确性至关重要，而对传递关系的稳健推理在因果推断等场景中尤为关键。因此，探究 Transformer 在推断传递关系（例如：已知 A 导致 B，B 导致 C，那么 A 导致 C）方面的能力十分必要。该任务等价于在有向图中判断连通性（例如：已知存在从 A 到 B 的路径，以及从 B 到 C 的路径，则存在从 A 到 C 的路径）。以往研究集中于 Transformer 是否能通过输入提示中的上下文示例学习传递性，但其在训练样本中学习传递关系的能力及规模扩展的影响仍不明确。本研究通过生成不同规模的有向图，训练多种规模的 Transformer 模型，并评估其对不同图结构的连通性推理能力。结果表明，Transformer 可以在‘网格状’有向图上学习连通性，其中每个节点可嵌入低维子空间，连通性可由节点嵌入轻松推断。底层网格图的维度是预测模型能力的关键因素，高维网格图比低维更难处理。此外，模型规模增大有助于提升在网格图上的泛化能力。然而，若图非网格结构且包含大量孤立连通分量，模型则难以学习连通性，尤其当组件数量增多时表现显著下降。
原文摘要 · Abstract (English)
Reasoning capability is essential to ensure the factual correctness of the responses of transformer-based Large Language Models (LLMs), and robust reasoning about transitive relations is instrumental in many settings, such as causal inference. Hence, it is essential to investigate the capability of transformers in the task of inferring transitive relations (e.g., knowing A causes B and B causes C, then A causes C). The task of inferring transitive relations is equivalent to the task of connectivity in directed graphs (e.g., knowing there is a path from A to B, and there is a path from B to C, then there is a path from A to C). Past research focused on whether transformers can learn to infer transitivity from in-context examples provided in the input prompt. However, transformers' capability to infer transitive relations from training examples and how scaling affects the ability is unexplored. In this study, we seek to answer this question by generating directed graphs to train transformer models of varying sizes and evaluate their ability to infer transitive relations for various graph sizes. Our findings suggest that transformers are capable of learning connectivity on "grid-like'' directed graphs where each node can be embedded in a low-dimensional subspace, and connectivity is easily inferable from the embeddings of the nodes. We find that the dimensionality of the underlying grid graph is a strong predictor of transformers' ability to learn the connectivity task, where higher-dimensional grid graphs pose a greater challenge than low-dimensional grid graphs. In addition, we observe that increasing the model scale leads to increasingly better generalization to infer connectivity over grid graphs. However, if the graph is not a grid graph and contains many disconnected components, transformers struggle to learn the connectivity task, especially when the number of components is large.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。