用查询与反例从Transformer中提取有限状态机，揭示其语言学习能力。
Extracting Moore Machines from Transformers using Queries and Counterexamples
- 通过查询与反例构建Transformer的高阶抽象——莫尔机。
- 在仅正例学习和序列准确率评估中验证方法有效性。
- 适合研究Transformer形式化能力的学者参考。
得益于变压器架构在深度学习中的流行，已有研究探讨了变压器从数据中可学习的形式语言。然而，由于方法学差异，现有结果难以比较。为此，我们利用查询与反例，从训练于正则语言的变压器中提取有限状态自动机作为高阶抽象。具体而言，我们提取莫尔机，因为文献中许多训练任务可映射为莫尔机。通过详细研究仅正例学习与序列准确率度量，我们展示了该方法的实用性。
原文摘要 · Abstract (English)
Fuelled by the popularity of the transformer architecture in deep learning, several works have investigated what formal languages a transformer can learn from data. Nonetheless, existing results remain hard to compare due to methodological differences. To address this, we construct finite state automata as high-level abstractions of transformers trained on regular languages using queries and counterexamples. Concretely, we extract Moore machines, as many training tasks used in literature can be mapped onto them. We demonstrate the usefulness of this approach by studying positive-only learning and the sequence accuracy measure in detail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。