动态分配推理计算资源，兼顾速度与成本。
Latency and Token-Aware Test-Time Compute
- 根据查询需求动态选择生成策略并分配计算量。
- 在推理任务中实现更高准确率，同时降低延迟和令牌开销。
- 适合需要快速响应的智能体应用，如自动规划与多步推理。
推理时扩展已成为提升大语言模型性能的有效方法，通过生成多个候选回复并从中选择最优。然而，现有动态计算分配研究多仅关注并行生成方法（如best-of-N），忽视了如束搜索等增量解码方法，且普遍忽略延迟问题，仅关注令牌使用量。本文将推理时扩展建模为动态计算分配与方法选择问题，系统需针对每个查询决定采用何种策略及分配多少计算资源。框架显式整合了令牌成本与实际运行时间（墙钟延迟），后者对用户体验至关重要，尤其在需高效发起多轮查询的智能体工作流中。在推理基准上的实验表明，该方法持续优于静态策略，在准确率-成本权衡上表现更优，且具备实际部署可行性。
原文摘要 · Abstract (English)
Inference-time scaling has emerged as a powerful way to improve large language model (LLM) performance by generating multiple candidate responses and selecting among them. However, existing work on dynamic allocation for test-time compute typically considers only parallel generation methods such as best-of-N, overlooking incremental decoding methods like beam search, and has largely ignored latency, focusing only on token usage. We formulate inference-time scaling as a problem of dynamic compute allocation and method selection, where the system must decide which strategy to apply and how much compute to allocate on a per-query basis. Our framework explicitly incorporates both token cost and wall-clock latency, the latter being critical for user experience and particularly for agentic workflows where models must issue multiple queries efficiently. Experiments on reasoning benchmarks show that our approach consistently outperforms static strategies, achieving favorable accuracy-cost trade-offs while remaining practical for deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。