让大模型在难决策时多算一点，提升效果还更快
HARP: Hesitation-Aware Reframing in Transformer Inference Pass
- 根据生成不确定性动态增加计算，模仿人类思考暂停与重构
- 在多个任务上提升最高达5.16%，且推理速度是束搜索的两倍
- 无需训练、兼容所有模型，适合想提速增效的研究者
本文针对大语言模型推理中不同令牌所需计算量差异的问题，提出HARP，一种对现成Transformer前向传播的简单改进。基于决策中的犹豫与框架效应，HARP在模型生成过程中遇到不确定时选择性增加计算。该方法模拟人类认知过程，在困难决策点暂停并重构输入以获得新视角。不同于其他方法，HARP具有模型无关性、免训练、易实现的特点。我们在多种下游任务和模型规模下评估该方法，结果显示性能提升最高达+5.16%。值得注意的是，其推理速度比束搜索快一倍。简单而有效，HARP揭示了自适应计算在提升Transformer类语言模型性能方面的潜力。
原文摘要 · Abstract (English)
This paper aims to improve the performance of large language models by addressing the variable computational demands in inference steps, where some tokens require more computational resources than others. We present HARP, a simple modification to "off-the-shelf" Transformer forward pass. Drawing from hesitation and the framing effect in decision-making, HARP selectively applies additional computation when the model encounters uncertainty during token generation. Our method mimics human cognitive processes by pausing at difficult decision points and reframing inputs for a different perspective. Unlike other approaches, HARP is model-agnostic, training-free, and easy to implement. We evaluate our method across various downstream tasks and model sizes, demonstrating performance improvements up to +5.16%. Notably, HARP achieves these gains while maintaining inference times twice faster than beam search. Simple and yet with significant gains, HARP provides insights into the potential of adaptive computation for enhancing the performance of Transformer-based language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。