用神经符号向量提升大模型的规则推理能力
Improving Rule-based Reasoning in LLMs using Neurosymbolic Representations
- 将隐藏状态转为神经符号向量，在符号空间中推理
- 数学推理任务正确率提升15.4倍，损失降低88.6%
- 适合需要高可靠规则推理的场景，如数学题求解
大型语言模型在解决需严格遵循规则的推理任务（如数学推理）时仍面临挑战。本文提出一种新型神经符号方法，将隐藏状态编码为神经符号向量，实现在神经符号向量空间中的问题求解。解码后的结果与原始隐藏状态融合，显著提升模型在数值推理任务上的表现。通过神经符号表示实现计算卸载，提升了效率、可靠性与可解释性。实验表明，相比思维链提示和监督微调（LoRA），该方法在一组数学推理任务上平均交叉熵损失降低88.6%，正确解题数提升15.4倍，且不影响其他任务性能。代码已开源：https://github.com/vdhanraj/Neurosymbolic-LLM。
原文摘要 · Abstract (English)
Large language models (LLMs) continue to face challenges in reliably solving reasoning tasks, particularly those that require precise rule following, as often found in mathematical reasoning. This paper introduces a novel neurosymbolic method that improves LLM reasoning by encoding hidden states into neurosymbolic vectors, enabling problem-solving within a neurosymbolic vector space. The results are decoded and merged with the original hidden state, significantly boosting the model's performance on numerical reasoning tasks. By offloading computation through neurosymbolic representations, this method enhances efficiency, reliability, and interpretability. Experimental results demonstrate an average of 88.6% lower cross-entropy loss and 15.4 times more problems correctly solved on a suite of mathematical reasoning tasks compared to chain-of-thought prompting and supervised fine-tuning (LoRA), without degrading performance on other tasks. We make our code available at: https://github.com/vdhanraj/Neurosymbolic-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。