在计算资源受限下，研究推理模型的安全性与能力平衡。
Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
- 用强化学习控制推理长度，实现可定制的思维链生成
- 量化策略提升计算效率，同时保持推理序列数量
- 揭示计算约束下安全性能与推理能力的权衡关系
测试时的计算扩展能通过生成更长的思维链（CoT）序列提升推理语言模型的表现，但随之带来显著的计算开销。本文研究两种计算约束策略：(1) 推理长度限制，(2) 模型量化，以降低推理模型的计算需求，并探究其对安全性的影响。具体而言，提出两种应用计算约束的方法：(1) 使用基于长度控制策略优化（LCPO）的强化学习方法微调推理模型，使其满足用户定义的思维链长度；(2) 应用量化技术，在用户设定的计算约束内最大化思维链序列生成量。此外，研究了计算效率与模型安全性之间的权衡关系。
原文摘要 · Abstract (English)
Test-time compute scaling has demonstrated the ability to improve the performance of reasoning language models by generating longer chain-of-thought (CoT) sequences. However, this increase in performance comes with a significant increase in computational cost. In this work, we investigate two compute constraint strategies: (1) reasoning length constraint and (2) model quantization, as methods to reduce the compute demand of reasoning models and study their impact on their safety performance. Specifically, we explore two approaches to apply compute constraints to reasoning models: (1) fine-tuning reasoning models using a length controlled policy optimization (LCPO) based reinforcement learning method to satisfy a user-defined CoT reasoning length, and (2) applying quantization to maximize the generation of CoT sequences within a user-defined compute constraint. Furthermore, we study the trade-off between the computational efficiency and the safety of the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。