用代码形式模拟状态追踪，发现线性RNN表现优于Transformer。
Learning State-Tracking from Code Using Linear RNNs
- 将排列组合转化为带打印的代码片段，实现状态可追踪训练
- 线性RNN在代码状态追踪任务中表现优异，而Transformer仍失败
- 揭示动作不可见时，非线性RNN反而更擅长状态追踪
近年来，状态追踪任务（尤其是排列组合）成为检验Transformer和RNN等序列模型能力的基准。然而这些任务通常是序列到序列的：学习将动作（排列）映射到状态，与语言模型常用的下一个词预测训练方式不兼容。为此，我们通过包含打印输出和变量变换的REPL追踪记录，将排列组合转换为代码形式，使状态信息得以显现。实验表明，具备状态追踪能力的线性RNN在此设置下表现良好，而Transformer依然失败。基于此表示，我们进一步研究代码中状态追踪为何普遍困难：动作并非总是完全可观测。我们将此建模为具有确定性状态揭示的随机有限状态自动机，发现在线性情况下，线性RNN甚至可能比非线性RNN更差。
原文摘要 · Abstract (English)
Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear). However, these are often sequence-to-sequence tasks: learning to map actions (permutations) to states, which is incompatible with the next-token prediction setting commonly used to train language models. We address this gap by converting permutation composition into code via REPL traces that interleave state-reveals through prints and variable transformations. We show that linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail. Motivated by this representation, we investigate why tracking states in code is generally difficult: actions are not always fully observable. We frame this as tracking the state of a probabilistic finite-state automaton with deterministic state reveals and show that linear RNNs can be worse than non-linear RNNs at tracking states in this setup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。