通过分层隐变量预测，提升大模型长程推理的准确性与效率。
Hierarchical Latent Prediction for Language Models
- 引入高层抽象隐变量，缓解隐空间多步推理的误差累积。
- 在代码生成和多步推理任务中实现更长时序的一致性表征。
- 适合需要高效推测解码的复杂推理场景，如编程与规划。
标准的下一个词预测（NTP）虽是语言模型预训练的基础，但其教师强制训练范式在长程推理和规划任务中可能并非最优。近期工作如多词预测（MTP）和下隐变量预测（NextLat）尝试通过预测多个未来词或在隐空间进行自监督预测来缓解该问题。然而，这些辅助目标或存在有限的预测时长，或因多步滚动导致误差累积。本文提出分层隐变量预测（HiLP），引入一个高层抽象隐变量，以减少隐空间滚动中的误差累积。实验表明，HiLP 能够实现更长时序的一致信念状态表征，在编码与多步推理基准测试中均表现优异，并显著提升推测解码效率。
原文摘要 · Abstract (English)
While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。