用简单Transformer破解密文，仅需1500个样本就可泛化到未见密钥。
ALICE: An Interpretable Neural Architecture for Generalization in Substitution Ciphers
- 采用仅编码器的Transformer架构，结合可解释的双射解码头。
- 在仅训练1500个密文后，准确率超越现有方法，且推理速度快。
- 揭示模型逐步从字母频次到词结构的推理过程，类比人类解密策略。
我们将密码破译作为研究神经网络推理与泛化的理想测试平台：模型需解密经替换密码编码的文本，从26!种可能映射中选择，且无法直接访问密钥。我们提出ALICE（用于学习可解释密文解密的架构），一个仅编码器的Transformer，在此解密任务上实现了新的准确率与速度基准。令人惊讶的是，ALICE在仅训练约1500个唯一密文后，即可泛化到未见过的密文，仅占全部密钥空间的3.7×10⁻²⁴。为增强可解释性，我们引入一种基于Gumbel-Sinkhorn方法的新型双射解码头，显式建模置换关系，可直接提取学习到的密钥映射。通过早停与探测实验，我们发现模型逐层优化预测——早期层更关注字母频率，后期层构建词级结构。这些架构创新与分析方法不仅适用于密文解密，也为神经网络泛化与可解释性提供了新洞见。
原文摘要 · Abstract (English)
We present cryptogram solving as an ideal testbed for studying neural network reasoning and generalization; models must decrypt text encoded with substitution ciphers, choosing from 26! possible mappings without explicit access to the cipher. We develop ALICE (an Architecture for Learning Interpretable Cryptogram dEcipherment), a simple encoder-only Transformer that sets a new state-of-the-art for both accuracy and speed on this decryption problem. Surprisingly, ALICE generalizes to unseen ciphers after training on only ${\sim}1500$ unique ciphers, a minute fraction ($3.7 \times 10^{-24}$) of the possible cipher space. To enhance interpretability, we introduce a novel bijective decoding head that explicitly models permutations via the Gumbel-Sinkhorn method, enabling direct extraction of learned cipher mappings. Through early exit and probing experiments, we reveal how ALICE progressively refines its predictions in a way that appears to mirror common human strategies -- early layers place greater emphasis on letter frequencies, while later layers form word-level structures. Our architectural innovations and analysis methods are applicable beyond cryptograms and offer new insights into neural network generalization and interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。