让Transformer用句子嵌入思考，用词元生成文本，兼顾语义抽象与训练稳定性。
SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens
- 用连续句子嵌入空间做推理，保留语义抽象能力。
- 在39M到1.3B参数规模下生成质量媲美主流模型。
- 无需扩散采样，恢复似然训练信号，适合研究生成机制的学者。
最近提出的大型概念模型（LCM）通过预测一系列句子级嵌入来生成文本，并采用均方误差或扩散目标进行训练。我们提出SONAR-LLM，一种仅解码器结构的Transformer，其‘思考’基于相同的连续SONAR嵌入空间，但通过冻结的SONAR解码器传播词元级别的交叉熵损失进行监督。该混合目标既保留了LCM的语义抽象性，又消除了其扩散采样过程，恢复了基于似然的训练信号。在从39M到1.3B参数的不同模型规模下，SONAR-LLM均实现了具有竞争力的生成质量。我们报告了缩放趋势、消融实验、基准测试结果，并发布完整的训练代码及所有预训练检查点，以促进可复现性和未来研究。
原文摘要 · Abstract (English)
The recently proposed Large Concept Model (LCM) generates text by predicting a sequence of sentence-level embeddings and training with either mean-squared error or diffusion objectives. We present SONAR-LLM, a decoder-only transformer that "thinks" in the same continuous SONAR embedding space, yet is supervised through token-level cross-entropy propagated via the frozen SONAR decoder. This hybrid objective retains the semantic abstraction of LCM while eliminating its diffusion sampler and restoring a likelihood-based training signal. Across model sizes from 39M to 1.3B parameters, SONAR-LLM attains competitive generation quality. We report scaling trends, ablations, benchmark results, and release the complete training code and all pretrained checkpoints to foster reproducibility and future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。