arXiv:2412.14076cs.AIcs.CL2024-12NeurIPS被引 8

提出可同时进行符号与神经计算的高效树结构模型,提升组合泛化能力。

Compositional Generalization Across Distributional Shifts with Sparse Tree Operations

  • 用稀疏向量表示符号结构,提升计算效率
  • 扩展至更通用的序列到序列任务,保持组合泛化能力
  • 全神经路径设计,避免符号计算主导的局限

神经网络在组合泛化方面仍存在困难,尤其缺乏大规模预训练时更为明显。虽然混合神经符号方法在实现类人组合泛化上取得成功,但其核心问题在于符号计算的可扩展性与灵活性不足。原因在于:这类模型将符号计算作为核心,仅用神经网络参数化符号系统。本文研究一种统一的神经符号系统,其中网络中的变换可同时解释为符号与神经计算。我们对名为 Differentiable Tree Machine 的统一神经符号架构进行两项关键改进:首先,通过稀疏向量表示符号结构显著提升效率;其次,将模型应用范围从受限的 tree2tree 问题扩展至更通用的 seq2seq 问题。新模型保留原有组合泛化能力,且因网络中存在完全神经路径,避免了其他神经符号技术中符号计算主导的问题。

原文摘要 · Abstract (English)

Neural networks continue to struggle with compositional generalization, and this issue is exacerbated by a lack of massive pre-training. One successful approach for developing neural systems which exhibit human-like compositional generalization is \textit{hybrid} neurosymbolic techniques. However, these techniques run into the core issues that plague symbolic approaches to AI: scalability and flexibility. The reason for this failure is that at their core, hybrid neurosymbolic models perform symbolic computation and relegate the scalable and flexible neural computation to parameterizing a symbolic system. We investigate a \textit{unified} neurosymbolic system where transformations in the network can be interpreted simultaneously as both symbolic and neural computation. We extend a unified neurosymbolic architecture called the Differentiable Tree Machine in two central ways. First, we significantly increase the model's efficiency through the use of sparse vector representations of symbolic structures. Second, we enable its application beyond the restricted set of tree2tree problems to the more general class of seq2seq problems. The improved model retains its prior generalization capabilities and, since there is a fully neural path through the network, avoids the pitfalls of other neurosymbolic techniques that elevate symbolic computation over neural computation.

神经符号组合泛化树结构序列生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。