对比概率电路与大模型表达能力，发现参数空间是关键瓶颈
The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models
- 将概率电路改用对数空间参数化，显著提升语言分布建模能力
- 固定路由结构限制了对复杂依赖关系的建模，导致性能下降
- 分解型电路比结构分解型更强大，但优化难度仍大
概率电路（PCs）是支持精确高效概率推断的深度生成模型。然而在自回归语言建模中，其表现仍远落后于基于Transformer的大语言模型（LLMs），表明存在显著的表达能力差距。本文在统一的自回归框架下比较了两类模型：首先，输出瓶颈在于PCs在概率空间中以凸组合方式参数化预测，难以捕捉语言中的尖锐分布；采用对数空间参数化可大幅缩小差距。其次，上下文编码瓶颈表现为：结构分解型PCs在与树形结构对齐的划分上可匹配Transformer的分离秩，但理论与实证均表明其能力受限于固定路由结构，当数据具有异构依赖拓扑时性能严重退化。进一步证明，分解型PCs严格优于结构分解型，但有效优化仍是开放挑战。
原文摘要 · Abstract (English)
Probabilistic Circuits (PCs) are deep generative models that support exact and efficient probabilistic inference. Yet in autoregressive language modeling, PCs still lag behind Transformer-based large language models (LLMs), suggesting an important expressivity gap. In this work, we compare PCs and LLMs under a unified autoregressive formulation. First, an output bottleneck: PCs parameterize predictions as convex combinations in probability space, which struggles to represent the sharp distributions typical of language; adopting a logit-space parameterization substantially narrows this gap. Second, a context-encoding bottleneck: we prove that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but show, both theoretically and empirically, that this capacity is limited to partitions aligned with the fixed routing structure, leading to severe degradation when the data exhibits heterogeneous dependency topologies. We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。