发现图注意力模型中拓扑结构是多跳推理的关键
What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs? A Study with Tabula RASA
- 通过消融实验分离出四种结构信号，发现稀疏邻接掩码主导性能提升
- 在3跳MetaQA上比无掩码模型高72.5个百分点，跨数据集均显著领先
- 适合关注知识图谱问答中模型设计与结构偏置的研究者
针对变换器在知识图谱上的多跳推理，我们通过控制性消融实验，对一个最小化变换器修改的四个可独立移除组件（稀疏邻接掩码、边类型偏置、查询缩放、值门控）进行分析，以隔离驱动推理的结构信号。结果清晰：仅稀疏邻接掩码就贡献了主要性能提升（3跳MetaQA上+72.5个百分点，WebQSP上+45.5，CWQ上+53.9），而学习到的关系参数仅带来小幅优化，且缺乏结构引导时反而有害。零样本实验进一步验证：当边类型被屏蔽时，基于掩码的注意力退化仅为关系特定权重的1/4。因此，多跳知识图谱问答的有效结构归纳偏置主要来自拓扑而非关系。
原文摘要 · Abstract (English)
What structural inductive bias helps transformers reason over knowledge graphs? Through controlled ablations of a minimal transformer modification with four independently removable components (sparse adjacency masking, edge-type biases, query scaling, value gating), we isolate which structural signals drive multi-hop reasoning. Our finding is sharp: sparse adjacency masking alone accounts for the dominant share of improvement over unmasked transformers (+72.5pp on 3-hop MetaQA, +45.5pp on WebQSP, +53.9pp on CWQ), while learned relation parameters add only modest refinement and can actively hurt without structural guidance. A zero-shot experiment provides architecturally independent corroboration: masking-based attention degrades 4.0x less than relation-specific weights when edge types are held out. The useful inductive bias for multi-hop KGQA is predominantly topological, not relational.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。