用可学习的字符串重写规则替代注意力,实现高效且可解释的序列生成。
RewriteNets: End-to-End Trainable String-Rewriting for Generative Sequence Modeling
- 通过可学习的规则进行并行字符串重写,显式建模序列结构
- 在SCAN基准上达到98.7%准确率,优于LSTM和Transformer
- 适合需要系统泛化能力的任务,如算法推理与字符串变换
主流序列模型如Transformer通过密集注意力隐式表示结构,导致二次复杂度。我们提出RewriteNets,一种基于显式并行字符串重写的新型神经架构。每个层包含一组可学习规则,对输入序列中每个位置执行四步操作:(1)规则模式的模糊匹配,(2)通过可微分分配算子解决冲突,选择不重叠的重写规则,(3)应用选定规则替换输入段为长度可变的输出段,(4)传递未被修改的标记。尽管规则分配是离散的,我们采用直通Gumbel-Sinkhorn估计器,实现稳定端到端训练。我们在算法、组合性及字符串操作任务上评估RewriteNets,对比强基准LSTM与Transformer。结果表明,RewriteNets在需要系统泛化任务中表现优异(在SCAN基准的长度分裂上达98.7%准确率),且计算效率高于Transformer。我们还分析了学习到的规则,并进行了详尽消融研究,证明该架构在引入显式结构归纳偏置方面具有前景。
原文摘要 · Abstract (English)
Dominant sequence models like the Transformer represent structure implicitly through dense attention weights, incurring quadratic complexity. We propose RewriteNets, a novel neural architecture built on an alternative paradigm: explicit, parallel string rewriting. Each layer in a RewriteNet contains a set of learnable rules. For each position in an input sequence, the layer performs four operations: (1) fuzzy matching of rule patterns, (2) conflict resolution via a differentiable assignment operator to select non-overlapping rewrites, (3) application of the chosen rules to replace input segments with output segments of potentially different lengths, and (4) propagation of untouched tokens. While the discrete assignment of rules is non-differentiable, we employ a straight-through Gumbel-Sinkhorn estimator, enabling stable end-to-end training. We evaluate RewriteNets on algorithmic, compositional, and string manipulation tasks, comparing them against strong LSTM and Transformer baselines. Results show that RewriteNets excel at tasks requiring systematic generalization (achieving 98.7% accuracy on the SCAN benchmark's length split) and are computationally more efficient than Transformers. We also provide an analysis of learned rules and an extensive ablation study, demonstrating that this architecture presents a promising direction for sequence modeling with explicit structural inductive biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。