根据问题难易动态分配计算资源，提升大模型推理效率与准确率
Predictive Scheduling for Efficient Inference-Time Reasoning in Large Language Models
- 用轻量预测器提前估算问题难度，决定生成长度
- 在GSM8K上相同算力下准确率提升7.9个百分点
- 适合对延迟和成本敏感的生产级大模型部署
大语言模型通过生成多条思维链（CoT）在复杂推理任务中取得领先性能，但固定每查询的令牌预算会导致简单输入过度计算、困难输入计算不足。我们提出预测调度（Predictive Scheduling），一种即插即用框架：预先运行轻量级预测器（MLP基于中间Transformer隐藏状态，或LoRA微调分类器基于原始问题文本），以估算每个查询的最佳推理长度或难度，再由贪心批处理分配器将固定总令牌预算动态分配至各查询，以最大化预期准确率。在GSM8K算术基准测试中，该方法在相同令牌成本下相比均匀预算可实现最高7.9个百分点的绝对准确率提升，关闭了超过50%与理想全知调度器之间的差距。层间系统性分析显示，Transformer中第12-17层包含最丰富的规模估计信号。结果表明，预运行预算预测可实现细粒度的算力-准确率权衡控制，为低延迟、低成本的大模型部署提供了切实路径。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve state-of-the-art accuracy on complex reasoning tasks by generating multiple chain-of-thought (CoT) traces, but using a fixed token budget per query leads to over-computation on easy inputs and under-computation on hard ones. We introduce Predictive Scheduling, a plug-and-play framework that pre-runs lightweight predictors, an MLP on intermediate transformer hidden states or a LoRA-fine-tuned classifier on raw question text, to estimate each query's optimal reasoning length or difficulty before any full generation. Our greedy batch allocator dynamically distributes a fixed total token budget across queries to maximize expected accuracy. On the GSM8K arithmetic benchmark, predictive scheduling yields up to 7.9 percentage points of absolute accuracy gain over uniform budgeting at identical token cost, closing over 50\% of the gap to an oracle with perfect foresight. A systematic layer-wise study reveals that middle layers (12 - 17) of the transformer carry the richest signals for size estimation. These results demonstrate that pre-run budget prediction enables fine-grained control of the compute-accuracy trade-off, offering a concrete path toward latency-sensitive, cost-efficient LLM deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。