通过分析模型隐状态动态,提升大模型推理中的探索效率。
ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
- 用柯普曼算子线性化隐状态动态,捕捉更丰富的计算结构。
- 提出动态谱分散度指标,量化模型潜空间探索程度。
- 在多模态和纯文本任务上均显著优于现有方法,适合强化学习优化研究者。
基于可验证奖励的强化学习(RLVR)最近在增强大推理模型(LRM)的推理能力方面展现出巨大潜力。然而,RLVR常导致策略过度确定性,引发无效探索和过早收敛。尽管提升标记层级多样性有助于缓解熵崩溃,我们指出,标记生成背后的潜在动态蕴含更丰富的计算结构,可用于引导策略优化实现更有效的探索-利用权衡。为实现对大推理模型潜动态的可处理分析与干预,我们采用柯普曼算子理论获得其隐藏状态动态的线性表示。这使我们引入动态谱分散度(DSD),一种量化模型潜动态异质性的新指标,作为策略探索的直接指示器。在此基础上,我们提出推理中的潜探索(ReLaX)框架,明确将潜动态纳入策略优化过程以调节探索与利用。在多种多模态及纯文本推理基准上的全面实验表明,ReLaX持续激励推理能力,并超越现有基于标记层级的方法。项目代码见 https://github.com/ZhangShimin1/ReLaX。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated remarkable potential in enhancing the reasoning capability of Large Reasoning Models (LRMs). However, RLVR often drives the policy toward over-determinism, resulting in ineffective exploration and premature policy convergence. While promoting token-level diversity has shown promise in mitigating entropy collapse, we argue that the latent dynamics underlying token generation encode a far richer computational structure for steering policy optimization toward a more effective exploration-exploitation tradeoff. To enable tractable analysis and intervention of the latent dynamics of LRMs, we leverage Koopman operator theory to obtain a linearized representation of their hidden state dynamics. This enables us to introduce Dynamic Spectral Dispersion (DSD), a new metric to quantify the heterogeneity of the model's latent dynamics, serving as a direct indicator of policy exploration. Building upon these foundations, we propose Reasoning with Latent eXploration (ReLaX), a framework that explicitly incorporates latent dynamics to regulate exploration and exploitation during policy optimization. Comprehensive experiments across a wide range of multimodal and text-only reasoning benchmarks show that ReLaX consistently incentivizes reasoning capability and outperforms existing token-level methods. Our project is available at https://github.com/ZhangShimin1/ReLaX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。