用矩阵非交换性编码词序，无需位置编码即可高效建模文本。
All You Need Is Non-Commutative Words
- 将词元表示为酉矩阵，通过矩阵乘积顺序捕获词序。
- 在IMDB上超越词袋模型，在AG News上表现相当，仅用64参数编码。
- 支持无投影自注意力与可并行处理变长文本，适合持续学习场景。
我们将词汇标记表示为酉矩阵，并将每个句子编码为其有序矩阵乘积。矩阵乘法的非交换性捕捉了词序，无需位置编码（PEs）。该代数结构还带来了多项能力：无需查询、键或值投影的反对称自注意力，以及以降低注意力成本实现变长文本块的并行组合。此外，它提供了一个规范-陪集读出层，紧凑地编码所有真实酉自由度，同时通过嵌套群扩张支持持续学习，每次新任务扩展算子空间但精确保留先前表示。在标准文本分类基准上，该方法匹配或超过词袋基线，在IMDB上实现更高准确率，在AG News上表现相当。值得注意的是，这是通过将传统约30,000维词汇空间替换为密集的64参数实值编码实现的，凸显了其参数化表达的高效性。
原文摘要 · Abstract (English)
We represent lexical tokens as unitary matrices and encode each sentence as their ordered product. The noncommutativity of matrix product captures word order without positional encodings (PEs). The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced attention cost. Furthermore, it provides a canonical-coset readout layer that encodes all true unitary degrees of freedom compactly, while supporting continual learning through nested group extensions that enlarge the operator space with each new task preserving prior representations exactly. Across standard text-classification benchmarks, the method matches or exceeds bag-of-words baselines. Achieving higher accuracy on IMDB and comparable performance on AG News. Notably, this is accomplished by replacing the conventional $\sim$30,000-dimensional vocabulary space with a dense, 64-parameter real-valued encoding, highlighting the expressive efficiency of our parameterization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。