arXiv:2506.07501cs.LGcs.AI2025-06

提出图式因果演化模型,解决大模型推理中长程依赖丢失问题

Graph-of-Causal Evolution: Challenging Chain-of-Model for Reasoning

  • 用可微稀疏因果邻接矩阵建模跨层依赖,通过因果掩码注意力传递约束
  • 在CLUTRR、CausalQA等数据集上超越链式模型,长程因果捕捉能力显著提升
  • 支持自演进机制,适合需持续优化的因果推理与复杂决策场景

针对链式模型(CoM)中各子链仅依赖前序信息、因果掩码阻断多层级间全局上下文流动导致长程依赖丢失的问题,本文提出图式因果演化(GoCE)模型。其核心思想是将隐式标记表示映射为可微稀疏的因果邻接矩阵,通过因果掩码注意力与因果MoE在每层计算中渗透因果约束。结合干预一致性损失测试与自演化门控机制,实现因果结构学习与Transformer架构自适应更新之间的动态平衡。实验在Claude Sonnet 4、o4-mini-high和DeepSeek R1构建的沙盒环境中进行,采用GoCE引入的Transformer变体架构,在CLUTRR、CLADDER、EX-FEVER和CausalQA等公开数据集上评估并对比基线大模型。结果表明,GoCE显著增强Transformer对长程因果依赖的捕捉能力,同时提升了模型自演进性能,不仅在设计原则上优于CoM,也为未来因果学习与持续自适应改进研究提供重要经验。

原文摘要 · Abstract (English)

In view of the problem that each subchain in the chain-of-model (CoM) relies only on the information of the previous subchain and may lose long-range dependencies due to the causal mask blocking the global context flow between multi-level subchains, this work proposes a graph of causal evolution (GoCE). Its core principle is to map the implicit token representation into a differentiable and sparse causal adjacency matrix, then permeate causal constraints through each layer of calculation using causal-masked attention and causal-MoE. By combining intervention consistency loss test and self-evolution gate, the dynamic balance between causal structure learning and adaptive updating of transformer architecture is realized. The researcher built experimental environments in sandboxes built with Claude Sonnet 4, o4-mini-high, and DeepSeek R1 respectively with the transformer variant architecture introduced in GoCE. It is evaluated on publicly available datasets including CLUTRR, CLADDER, EX-FEVER, and CausalQA and compared with the baseline LLMs. The finding proves that GoCE strengthens the transformer's ability to capture long-range causal dependencies, while the ability to self-evolve is improved. It not only surpasses the design of CoM in terms of design principles, but also provides experience for future research on causal learning and continuous adaptive improvement.

因果推理自演化长程依赖Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。