通过只保留起始和候选答案令牌,提升大模型早期预测准确性。
SimLens for Early Exit in Large Language Models: Eliciting Accurate Latent Predictions with One More Token
- 仅保留起始符和候选答案符,轻量续推实现高精度隐状态预测。
- 在ARC、BoolQ等数据集上,平均准确率提升0.43,计算开销可控。
- 适合追求高效推理的场景,尤其适用于低延迟部署的大模型应用。
大型语言模型(LLMs)的中间层预测具有信息量但难以准确解码,尤其是在早期层。现有基于镜头的方法通常依赖直接线性读出,虽简单却常偏离最终预测。我们提出SimLens,一种无需训练的单令牌决策解码器,仅保留起始令牌([s])和候选答案令牌([a]),并执行一次轻量级后续前向传播。这一微小改动显著提升了隐状态预测的准确性。我们进一步引入Linear SimLens,用于基于熵的置信度估计,并将两者结合形成SimExit混合早退机制。在使用LLaMA-7B和Vicuna-7B的ARC、BoolQ和HeadQA数据集上,SimLens在所有六个设置中均提升等计算量下的准确率,平均增益达+0.43,即使包含额外两个令牌的后向开销也成立。SimExit在最优准确率点实现平均1.15倍加速,在允许最多1个百分点精度下降时达到1.40倍加速。消融实验表明,[s]与[a]分别扮演全局条件与语义锚点角色。
原文摘要 · Abstract (English)
Intermediate-layer predictions in large language models (LLMs) are informative but hard to decode accurately, especially at early layers. Existing lens-style methods typically rely on direct linear readout, which is simple but often drifts away from the model's eventual prediction. We proposeSimLens, a simple training-free decoder for single-token decision tasks that keeps only the start token and a candidate answer token ([s] and [a]) and performs one lightweight continuation through the remaining upper layers. This surprisingly small modification recovers much more accurate latent predictions than direct linear decoding. We further introduce Linear SimLens, a lightweight linear approximation for entropy-based confidence estimation, and combine the two in SimExit, a hybrid early-exit mechanism. On ARC, BoolQ, and HeadQA with LLaMA-7B and Vicuna-7B, SimLens improves Iso-Compute accuracy in all six settings, with an average gain of +0.43 even when fair compute includes the extra two-token post-forward overhead. SimExit yields an average 1.15$\times$ speedup at the best-accuracy operating points and 1.40$\times$ when allowing up to a 1 percentage-point accuracy drop. Ablations show that [s] and [a] play distinct roles as global condition and semantic anchor, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。