用连续概念提升大模型预训练效率与可解释性
LLM Pretraining with Continuous Concepts
- 将连续概念注入隐藏状态，融合离散文本预测
- 在多个任务上样本效率更高，性能优于传统方法
- 支持直接修改概念，增强模型推理透明度
当前大语言模型预训练普遍采用下一词预测目标，通过优化词级困惑度学习表征。本文提出一种新框架 Continuous Concept Mixing（CoCoMix），将离散的下一词预测与从预训练稀疏自编码器中学习到的连续概念相结合。CoCoMix 通过交错方式将连续概念混入模型隐藏状态，实现端到端联合优化。在多种基准测试（包括语言建模和下游推理任务）中，CoCoMix 表现更优，样本效率更高，显著超越标准下一词预测、知识蒸馏及插入暂停标记的方法。实验表明，概念学习与交错机制协同作用对性能提升至关重要。此外，该方法提升了模型可解释性与可控性，支持直接观察和修改预测概念，为引导模型内部推理过程提供透明路径。
原文摘要 · Abstract (English)
Next token prediction has been the standard training objective used in large language model pretraining. Representations are learned as a result of optimizing for token-level perplexity. We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts. Specifically, CoCoMix predicts continuous concepts learned from a pretrained sparse autoencoder and mixes them into the model's hidden state by interleaving with token hidden representations. Through experiments on multiple benchmarks, including language modeling and downstream reasoning tasks, we show that CoCoMix is more sample efficient and consistently outperforms standard next token prediction, knowledge distillation and inserting pause tokens. We find that combining both concept learning and interleaving in an end-to-end framework is critical to performance gains. Furthermore, CoCoMix enhances interpretability and steerability by allowing direct inspection and modification of the predicted concept, offering a transparent way to guide the model's internal reasoning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。