用向量符号处理融合对象与关系特征,提升模型推理效率与准确率。
RESOLVE: Relational Reasoning with Symbolic and Object-Level Features Using Vector Symbolic Processing
- 通过向量符号运算实现对象与关系特征共存,避免相互干扰。
- 在排序和数学题求解任务中,准确率优于现有方法,延迟更低。
- 采用双极高维空间注意力机制,计算更快,适合实时推理场景。
基于Transformer的编码器-解码器架构在推理任务中表现受限,因其难以有效提取输入对象间的关系信息。尽管已有研究引入抽象模块(Abstractor)以增强关系建模能力,但该模块仅擅长纯关系推理,在需要同时处理对象与关系的任务中表现不佳。为此,我们提出RESOLVE——一种神经-向量符号架构,通过高维空间中的捆绑(求和)与绑定(Hadamard积)操作,将对象级特征与关系表示融合,实现二者共存且互不干扰。模型采用新型双极高维空间注意力机制,显著降低注意力计算延迟。实验表明,RESOLVE在纯关系推理任务(如排序)及部分关系推理任务(如数学问题求解)中均达到更高准确率,同时具备更低计算延迟与内存开销,展现出更强泛化能力。
原文摘要 · Abstract (English)
Modern transformer-based encoder-decoder architectures struggle with reasoning tasks due to their inability to effectively extract relational information between input objects (data/tokens). Recent work introduced the Abstractor module, embedded between transformer layers, to address this gap. However, the Abstractor layer while excelling at capturing relational information (pure relational reasoning), faces challenges in tasks that require both object and relational-level reasoning (partial relational reasoning). To address this, we propose RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces, using fast and efficient operations such as bundling (summation) and binding (Hadamard product) allowing both object-level features and relational representations to coexist within the same structure without interfering with one another. RESOLVE is driven by a novel attention mechanism that operates in a bipolar high dimensional space, allowing fast attention score computation compared to the state-of-the-art. By leveraging this design, the model achieves both low compute latency and memory efficiency. RESOLVE also offers better generalizability while achieving higher accuracy in purely relational reasoning tasks such as sorting as well as partial relational reasoning tasks such as math problem-solving compared to state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。