Transformer通过有限访问序列处理实现组合泛化，准确率接近100%。
Exploring Compositional Generalization (in COGS/ReCOGS_pos) by Transformers using Restricted Access Sequence Processing (RASP)
- 用RASP框架设计扁平模式匹配规则，仅靠词级标记和一次注意力计算
- 在COGS与ReCOGS_pos任务上达到近满分的结构与语义精确匹配
- 无需递归树结构，适合研究组合泛化机制的学者参考
人类能理解新出现的词语组合，这种能力称为组合泛化。COGS基准（Kim and Linzen, 2020）显示，变压器模型在某些结构泛化任务上准确率为0%。本文使用受限访问序列处理（RASP）框架，证明编码器-解码器型变压器可系统性地完成COGS及语义等价的ReCOGS_pos（Wu et al., 2024）任务：模型在结构泛化测试集上取得精确匹配（exact match）或语义精确匹配（semantic exact match）近满分成绩。研究发现，(Re)COGS任务并不要求层次化或树状结构解法。模型采用词级标记与一次嵌入层标注可能词性，配合19个兼容注意力头的扁平模式匹配规则（可通过特定训练样例识别），结合语法覆盖率（Zeller et al., 2023）验证其覆盖输入语法的非递归部分；在识别句中与主谓相关名词时，通过掩码处理介词短语（pp noun）和从句补足成分（cp）。解码器循环生成逻辑形式（LF）token直至完成，不依赖如‘np_det pp np → np_pp → np’这类递归树形规则。
原文摘要 · Abstract (English)
Humans understand new combinations of words encountered if they are combinations of words recognized from different contexts, an ability called Compositional Generalization. The COGS benchmark (Kim and Linzen, 2020) arXiv:2010.05465 reports 0% accuracy for Transformer models on some structural generalizations. We use (Weiss et al., 2021) arXiv:2106.06981's Restricted Access Sequence Processing (RASP), a Transformer-equivalent programming language, to demonstrate that a Transformer Encoder-Decoder can perform COGS and the semantically equivalent ReCOGS_pos (Wu et al., 2024) arXiv:2303.13716 systematically and compositionally: Our RASP models attain near perfect scores on structural generalization splits on COGS (exact match) and ReCOGS_pos (semantic exact match). Our RASP models show the (Re)COGS tasks do not require a hierarchical or tree-structured solution (contrary to (Kim and Linzen, 2020) arXiv:2010.05465, (Yao and Koller, 2022) arXiv:2210.13050, (Murty et al., 2022) arXiv:2211.01288, (Liu et al., 2021) arXiv:2107.06516): we use word-level tokens with an "embedding" layer that tags with possible part of speech, applying just once per encoder pass 19 attention-head compatible flat pattern-matching rules (easily identified with specific training examples), shown using grammar coverage (Zeller et al., 2023) to cover the non-recursive aspects of the input grammar, plus masking out prepositional phrases ("pp noun") and/or sentential complements (cp) when recognizing grammar patterns and extracting nouns related to the main verb in the sentence, and output the next logical form (LF) token (repeating until the LF is complete). The models do not apply recursive, tree-structured rules like "np_det pp np -> np_pp -> np", but score near perfect semantic and string exact match on both COGS and ReCOGS pp recursion, cp recursion using the decoder loop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。