arXiv:2506.11077cs.CL2025-06中稿 · ICLR被引 7

通过周期性调度反思标记,提升大模型推理性能。

CyclicReflex: Improving Reasoning Models via Cyclical Reflection Token Scheduling

  • 提出周期性反思标记调度策略,动态调节反思时机与频率。
  • 在多个数据集上实现1.2%~4.7%的准确率提升,优于现有方法。
  • 无需额外计算,适合各类大小推理模型快速部署。

大型推理模型(LRMs)如OpenAI的o1和DeepSeek-R1利用测试时扩展进行多步推理以解决复杂问题。该推理过程通过特殊转折标记(如“wait”、“but”、“alternatively”)触发自我评估式反思,这些标记统称为“反思标记”。本文将反思标记视为一种资源,提出资源分配问题,旨在通过自适应调节其频率与位置来提升测试时计算效率。实验表明,过度或不足使用反思标记(即过反思与欠反思)均会降低模型表现。基于此,类比优化中的学习率调度,提出无需训练的解码策略CyclicReflex:采用双向、位置相关的三角波形动态调节反思标记的逻辑值,不增加额外计算开销。在MATH500、AIME2024/2025、AMC2023、GPQA Diamond和LiveCodeBench上的实验显示,该方法在1.5B至14B规模模型上均持续提升性能,超越标准解码及TIP、S1等近期方法。代码已开源。

原文摘要 · Abstract (English)

Large reasoning models (LRMs), such as OpenAI's o1 and DeepSeek-R1, harness test-time scaling to perform multi-step reasoning for complex problem-solving. This reasoning process, executed before producing final answers, is often guided by special juncture tokens that prompt self-evaluative reflection. These transition markers and reflective cues are referred to as "reflection tokens" (e.g., "wait", "but", "alternatively"). In this work, we treat reflection tokens as a "resource" and introduce the problem of resource allocation, aimed at improving the test-time compute performance of LRMs by adaptively regulating the frequency and placement of reflection tokens. Through empirical analysis, we show that both excessive and insufficient use of reflection tokens, referred to as over-reflection and under-reflection, can degrade model performance. To better understand this trade-off, we draw an analogy between reflection token usage and learning rate scheduling in optimization. Building on this insight, We propose cyclical reflection token scheduling (termed CyclicReflex), a training-free decoding strategy that dynamically modulates reflection token logits with a bidirectional, position-dependent triangular waveform, incurring no additional computation cost. Experiments on MATH500, AIME2024/2025, AMC2023, GPQA Diamond and LiveCodeBench demonstrate that CyclicReflex consistently improves performance across model sizes (1.5B-14B), outperforming standard decoding and recent approaches such as TIP (thought switching penalty) and S1. Codes are available at https://github.com/OPTML-Group/CyclicReflex.

推理增强解码策略反思机制优化调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。