研究大模型在输出长度受限下的推理能力，发现最佳模型和提示策略会随约束变化。
An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint
- 测试30个大模型在不同输出长度限制下的推理表现
- 发现模型大小和提示风格的最优选择随预算变化而改变
- 为实际部署中低延迟场景提供可操作的选型建议
近期研究展示了大语言模型（LLMs）在推理时扩展的巨大潜力：通过思考后再回答，模型可在额外推理计算下显著提升准确率。然而，在许多真实场景中，模型需在时间约束下快速响应，即答案必须在特定输出长度内生成。目前尚不清楚不同LLMs在严格约束下的推理能力是否依然有效。本文首次深入探讨该问题，对30个主流LLMs在多个常见推理数据集上进行系统性评估，覆盖广泛的输出长度预算，并分析准确率与模型类型、规模、提示风格等属性之间的关联。同时，研究还考察了令牌预算与实际设备延迟预算之间的映射关系。结果揭示了若干与无约束情况不同的有趣现象，例如在不同预算下，最优模型规模或提示风格会发生变化。这些发现为该领域提供了及时的评估基准，并为用户在真实延迟约束下部署模型提供了实用指导。
原文摘要 · Abstract (English)
Recent work has demonstrated the remarkable potential of Large Language Models (LLMs) in test-time scaling. By making models think before answering, they are able to achieve much higher accuracy with extra inference computation. However, in many real-world scenarios, models are used under time constraints, where an answer should be given within a certain output length. It is unclear whether and how the reasoning ability of different LLMs remain effective under strict constraints. We take a first look at this problem by conducting an in-depth empirical study. Specifically, we test 30 LLMs on common reasoning datasets under a wide range of output length budgets, and we analyze the correlation between the inference accuracy and various properties including model type, model size, prompt style, etc. We also consider the mappings between token budgets and actual on-device latency budgets. The results have demonstrated several interesting findings regarding the budget-aware LLM reasoning ability that differ from the unconstrained situation, e.g. the optimal choices of either model size or prompt style change under different budgets. These findings offer timely evaluation to this area and practical guidance for users to deploy LLMs under real-world latency constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。