让语言模型学概念而非单个词，提升语义对齐与下游任务表现
Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models
- 用语义等价词集替代单个目标词，实现自监督概念学习
- 在分类、聚类、重排序任务上性能提升，且低困惑度下保持语言质量
- 适合追求语义理解与模型泛化能力的研究者
传统下一个词预测（NTP）目标要求模型在每一步预测单一词元，尽管许多延续可表达相同含义。例如，在句子“this sticker can be placed here”中，“positioned”“attached”或“put”都是合理选择。标准NTP训练将这些选项视为互斥目标，而本文提出一种自监督框架，鼓励模型预测概念，即由语义等价词组成的集合。经概念监督训练的模型更符合人类相似性判断，提升分类、聚类和重排序性能，并在下游推理任务中达到相当或更强表现。该方法在语义相关词上降低困惑度（第3.2节），全局困惑度仅小幅上升，表明概念学习能增强语义对齐同时保持语言建模质量。代码已公开于https://github.com/christine-zhang1/learning-concepts。
原文摘要 · Abstract (English)
The next-token prediction (NTP) objective trains language models to predict a single token at each step, even though many continuations can express the same meaning. For example, in the sentence ``this sticker can be placed here'', positioned, attached, or put are all plausible alternatives. While standard NTP training treats these alternatives as mutually exclusive targets, we explore a self-supervised framework that encourages models to predict concepts, approximated as sets of semantically equivalent tokens. Models trained with this concept supervision align better with human similarity judgments, improve classification, clustering, and reranking performance, and achieve comparable or stronger downstream reasoning. These gains come with lower perplexity on semantically meaningful words (Section 3.2) and only minimal increases in global perplexity, suggesting that concepts enhance semantic alignment while preserving language modeling quality. Our code is available at https://github.com/christine-zhang1/learning-concepts
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。