用几何基元重构线性层，参数量降为原来的1/log²d
Composing Linear Layers from Irreducibles
- 基于克利福德代数将线性层分解为旋转向量的乘积
- 仅需O(log²d)参数即可逼近稠密矩阵的O(d²)能力
- 适用于大模型注意力机制，性能媲美主流压缩方法
当前大型模型常表现出由底层基础单元组合而成的特性，但这些基本构建块仍不明确。本文研究线性层中的这种组合结构：能否从一组最小几何基元中识别或合成线性变换？通过克利福德代数，我们证明线性层可表示为双矢量（编码有向平面的几何对象）的组合，并提出一种可微算法将其分解为旋转向量的乘积。该构造仅需O(log² d)参数，远少于稠密矩阵所需的O(d²)。在大语言模型注意力机制的键、查询和值投影中应用，基于旋转向量的层性能与块哈达玛德及低秩近似等强基线相当。研究结果为深度模型中几何基元如何组合形成高层功能提供了代数视角。
原文摘要 · Abstract (English)
Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. We investigate this compositional structure in linear layers by asking: can we identify/synthesize linear transformations from a minimal set of geometric primitives? Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors -- geometric objects encoding oriented planes -- and introduce a differentiable algorithm that decomposes them into products of rotors. This construction uses only O(log^2 d) parameters, versus O(d^2) required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。