让语言模型学会识别可替换的符号，提升逻辑推理能力。
Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence
- 用共享+随机双组件嵌入，平衡语义一致与符号区分
- 在扩展词汇下任务准确率提升12%-18%，尤其在逻辑推理中
- 适合需要形式化推理、符号重命名不变性的研究者
语言模型缺乏可替换符号的概念：语义等价但形式不同的符号（如形式逻辑中的绑定变量）。这一局限导致其难以泛化到更大词汇表，也难以识别α等价性——即变量重命名不改变含义。我们形式化该问题，提出α协方差度量以评估模型对这类变换的鲁棒性。为此，我们设计双组件词嵌入策略：共享部分保证语义一致性，随机部分维持符号可区分性。相比依赖α重命名的数据增强基线，我们的方法在线性时序逻辑求解、命题逻辑赋值预测和可扩展词汇复制任务中均实现更好泛化性能，且引入有利于α等价性的归纳偏置。结果为构建能学习可替换符号表示的语言模型奠定基础，是迈向更灵活系统化形式推理的关键一步。代码与项目页见https://necrashter.github.io/interchangeable-token-embeddings
原文摘要 · Abstract (English)
Language models lack the notion of interchangeable tokens: symbols that are semantically equivalent yet distinct, such as bound variables in formal logic. This limitation prevents generalization to larger vocabularies and hinders the model's ability to recognize alpha-equivalence, where renaming bound variables preserves meaning. We formalize this machine learning problem and introduce alpha-covariance, a metric for evaluating robustness to such transformations. To tackle this task, we propose a dual-part token embedding strategy: a shared component ensures semantic consistency, while a randomized component maintains token distinguishability. Compared to a baseline that relies on alpha-renaming for data augmentation, our approach demonstrates improved generalization to unseen tokens in linear temporal logic solving, propositional logic assignment prediction, and copying with an extendable vocabulary, while introducing a favorable inductive bias for alpha-equivalence. Our findings establish a foundation for designing language models that can learn interchangeable token representations, a crucial step toward more flexible and systematic reasoning in formal domains. Our code and project page are available at https://necrashter.github.io/interchangeable-token-embeddings
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。