让模型无视符号名字变化，实现真正意义上的开放词汇学习。
Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning
- 用多路嵌入分离可互换符号贡献，结合聚合注意力共享结构信息。
- 在未见过的符号上表现显著优于传统方法，验证了理论不变性。
- 适合需要符号泛化的逻辑推理、程序生成等任务场景。
当前神经网络缺乏处理可互换标记（如绑定变量）的合理机制，导致在固定词表上训练的模型难以泛化到未见符号，即使语义保持不变。本文提出一种新型Transformer机制，可严格保证对可互换标记重命名的不变性。该方法采用并行嵌入流，分离每个可互换标记的输入贡献，并通过聚合注意力机制实现跨流的结构化信息共享。实验结果验证了理论保证，并在需泛化到新符号的开放词汇任务中取得显著性能提升。
原文摘要 · Abstract (English)
Current neural architectures lack a principled way to handle interchangeable tokens, i.e., symbols that are semantically equivalent yet distinguishable, such as bound variables. As a result, models trained on fixed vocabularies often struggle to generalize to unseen symbols, even when the underlying semantics remain unchanged. We propose a novel Transformer-based mechanism that is provably invariant to the renaming of interchangeable tokens. Our approach employs parallel embedding streams to isolate the contribution of each interchangeable token in the input, combined with an aggregated attention mechanism that enables structured information sharing across streams. Experimental results confirm the theoretical guarantees of our method and demonstrate substantial performance gains on open-vocabulary tasks that require generalization to novel symbols. Project page: https://bu-depend-lab.github.io/Symbol-Invariant-Transformer/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。