用思维链揭示布尔函数的复杂度,链接了学习理论与Transformer结构。
Ehrenfeucht-Haussler Rank and Chain of Thought
- 将函数秩对应为单层Transformer所需思维链步数
- 证明ℓ重复合函数需恰好ℓ步思维链,定位第k个1需k步
- 提出多头秩概念,分析有界多头秩函数的可学习性
布尔函数的‘秩’是概率近似正确(PAC)学习理论中的核心概念,支持多项式大小决策树的准多项式时间学习算法。本文基于著名的Transformer架构,提出秩的新表征:函数f的秩等于单层硬注意力Transformer实现f所需的最少思维链(Chain of Thought, CoT)步骤数。基于该表征,我们建立了特定问题的思维链步数紧致上界,证明ℓ重函数复合恰好需要ℓ个CoT步骤;同时,分析布尔序列中第k个1的位置识别问题,证明其需恰好k个CoT步骤。最后,引入多头秩概念以刻画多头单层Transformer,并研究有界多头秩函数类的PAC可学习性。
原文摘要 · Abstract (English)
The notion of \emph{rank} of a Boolean function has been a cornerstone in PAC learning theory, enabling quasipolynomial-time learning algorithms for polynomial-size decision trees. We present a novel characterization of rank, grounded in the well-known Transformer architecture. We show that the rank of a function $f$ corresponds to the minimum number of \emph{Chain of Thought} (CoT) steps required by a single-layer Transformer with hard attention to compute $f$. Based on this characterization we establish tight bounds on the number of CoT steps required for specific problems, showing that \(\ell\)-fold function composition necessitates exactly \(\ell\) CoT steps. Furthermore, we analyze the problem of identifying the position of the \(k\)-th occurrence of 1 in a Boolean sequence, proving that it requires \(k\) CoT steps. Finally, we introduce the notion of the multi-head rank that captures multi-head single-layer transformers, and perform the analysis of PAC-learnability of the classes of functions with bounded multi-head rank.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。