用因果概念引导扩散模型，提升语言推理能力
C$^2$DLM: Causal Concept-Guided Diffusion Large Language Models
- 从教师模型提取概念因果图，指导注意力学习因果关系
- 在推理任务上提升12%，训练速度加快3.2倍
- 适合需要强逻辑推理的AI研究者和开发者
自回归语言模型与扩散语言模型（DLM）是大语言模型的两大主流范式，但均存在推理能力不足的问题。人类推理依赖因果知识与思维，自然语言中也蕴含灵活的因果结构。而自回归模型将语言建模为严格左到右的词元预测，扩散模型则采用全连接注意力，完全忽略因果顺序。为此，本文提出因果概念引导的扩散语言模型（C²DLM）。该模型首先从教师模型中获取概念级因果图，再显式引导注意力学习概念间的因果关系。通过聚焦因果关联并避免因果反转带来的干扰，C²DLM在COT-OrderPerturb任务上性能提升12%，训练速度提高约3.2倍，并在六个下游推理任务上平均提升1.31%。
原文摘要 · Abstract (English)
Autoregressive (AR) language models and Diffusion Language Models (DLMs) constitute the two principal paradigms of large language models. However, both paradigms suffer from insufficient reasoning capabilities. Human reasoning inherently relies on causal knowledge and thought, which are reflected in natural language. But in the AR paradigm, language is modeled as next token prediction (a strictly left-to-right, token-by-token order), whereas natural language itself exhibits more flexible causal structures. In the DLM paradigm, the attention mechanism is fully connected, which entirely disregards causal order. To fill this gap, we propose a \underline{\textbf{C}}ausal \underline{\textbf{C}}oncept-Guided \underline{\textbf{D}}iffusion \underline{\textbf{L}}anguage \underline{\textbf{M}}odel (C$^2$DLM). Starting from DLM's fully connected attention, C$^2$DLM first obtains a concept-level causal graph from the teacher model, and then explicitly guides attention to learn causal relationships between concepts. By focusing on causal relationships and avoiding interference from difficult subgoals involving causal inversion, C$^2$DLM improves 12\% with about 3.2 times training speedup in the COT-OrderPerturb task, and achieves an average gain of 1.31\% across six downstream reasoning tasks. More details in the repository ~\href{https://github.com/Kairong-Han/C-2-DLM}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。