为大模型推理中的预算浪费问题提供动态分配方案
On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiency
- 基于模型隐状态预测每个样本最优计算预算
- 实验显示预算利用率提升超40%且不损失整体准确率
- 适合追求推理效率的部署场景，尤其关注延迟与内存
并行思维通过多路径采样与聚合提升大模型推理能力。标准评估中因缺乏样本特定先验，所有样本共享全局预算以最大化数据集准确率。然而许多样本在远小于全局预算时已达最佳准确率，导致预算利用率低下。这种系统效能与样本效率的矛盾构成‘过度扩展诅咒’。本文首次对这一现象进行形式化分析，并量化其在真实系统中的普遍性与严重性。为此提出潜空间预算预测器（LanBo），通过探测模型隐状态预测样本级最优预算。LanBo显著提升预算利用率且保持数据集准确率。进一步将LanBo融入完整解码流程，提出预解码预算自适应（PreAda）范式，在解码前分配预算，保留并行性。实验表明，该方法大幅优化延迟与内存使用，展现其在硬件感知效率上的实用价值及对高效并行解码的潜力。
原文摘要 · Abstract (English)
Parallel thinking improves LLM reasoning through multi-path sampling and aggregation. In standard evaluations, due to a lack of sample-specific priors, all samples share a global budget chosen to maximize dataset accuracy. However, many samples reach their best accuracy with much smaller budgets, causing low budget utilization. This contradiction between system efficacy and sample efficiency constitutes the Overscaling Curse. In this paper, we first provide a formal analysis of the overscaling curse and quantify its prevalence and severity in real-world systems. To break it, we propose Latent Budget Predictor (LanBo), which probes model latent representations to predict sample-specific optimal budgets. LanBo significantly improves budget utilization while maintaining dataset accuracy. We further integrate LanBo into the full decoding pipeline, inspiring Pre-decoding Budget Adaptation (PreAda), a paradigm that allocates budgets before decoding to preserve decoding-time parallelization. LanBo substantially improves hardware-aware efficiency in latency and memory, demonstrating both its practical value and the promise of LanBo for efficient parallel decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。