用酉群子群构建新型RNN与Transformer，提升序列建模性能
Subgroups of $U(d)$ Induce Natural RNN and Transformer Architectures
- 基于酉群子群设计统一的序列模型框架
- 正交状态模型在两个数据集上表现优于参数匹配基线
- 共享骨架支持灵活替换状态空间，适合理论研究者
本文提出一种直接框架，用于在U(d)的闭合子群上构建带隐藏状态的序列模型。通过最小公理设定，从共享骨架推导出循环神经网络和Transformer模板，其中子群选择可替代状态空间、切向投影和更新映射。我们聚焦于O(d)，在参数匹配条件下对正交状态RNN和Transformer在Tiny Shakespeare与Penn Treebank上进行评估。此外，报告了一种通用的切向空间线性混合扩展，适用于所有子群选择，并在当前的O(d)实验中提升了有限预算下的性能。
原文摘要 · Abstract (English)
This paper presents a direct framework for sequence models with hidden states on closed subgroups of U(d). We use a minimal axiomatic setup and derive recurrent and transformer templates from a shared skeleton in which subgroup choice acts as a drop-in replacement for state space, tangent projection, and update map. We then specialize to O(d) and evaluate orthogonal-state RNN and transformer models on Tiny Shakespeare and Penn Treebank under parameter-matched settings. We also report a general linear-mixing extension in tangent space, which applies across subgroup choices and improves finite-budget performance in the current O(d) experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。