arXiv:2512.17206cs.CV2025-12被引 2

通过潜变量控制大模型推理路径,实现可控探索与高效学习。

Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs

  • 用变分自编码器生成潜变量,引导模型在生成前规划不同推理策略。
  • 在多个推理基准上表现优于标准强化学习方法,提升探索效率。
  • 适合需要可控、多样推理输出的研究者或应用开发者。

探索能力影响大型(视觉)语言模型的推理性能和强化学习训练效果,因随机采样常导致冗余推理路径且缺乏高层多样性。本文提出「推理调色板」(Reasoning Palette),一种基于潜变量调节的框架,为模型引入随机潜变量以实现战略性上下文构建,在生成令牌前引导内部规划。该潜变量由问题-答案对的均值池化嵌入通过变分自编码器(VAE)推断,每次采样可能编码不同的推理语境。推理时,采样潜变量解码为可学习的令牌前缀,并前置到输入提示中,从而调节模型的内部推理轨迹。模型在输出生成前完成对推理策略的内部采样,塑造响应序列的整体风格与结构。通过简短的监督微调(SFT)预热阶段,模型可适应此潜变量条件。在强化学习优化中,该方法通过按需注入多样化推理模式,实现结构化探索,显著提升探索效率与持续学习能力。多基准实验表明,本方法能对(视觉)语言模型的战略行为进行可解释、可控的调控,一致优于标准强化学习方法。

原文摘要 · Abstract (English)

Exploration capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models, as stochastic sampling often yields redundant reasoning paths with little high-level diversity. This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a stochastic latent variable for strategic contextualization, guiding its internal planning prior to token generation. This latent context is inferred from the mean-pooled embedding of a question-answer pair via a variational autoencoder (VAE), where each sampled latent potentially encodes a distinct reasoning context. During inference, a sampled latent is decoded into learnable token prefixes and prepended to the input prompt, modulating the model's internal reasoning trajectory. In this way, the model performs internal sampling over reasoning strategies prior to output generation, which shapes the style and structure of the entire response sequence. A brief supervised fine-tuning (SFT) warm-up phase allows the model to adapt to this latent conditioning. Within RL optimization, Reasoning Palette facilitates structured exploration by enabling on-demand injection for diverse reasoning modes, significantly enhancing exploration efficiency and sustained learning capability. Experiments across multiple reasoning benchmarks demonstrate that our method enables interpretable and controllable control over the (vision-) language model's strategic behavior, thereby achieving consistent performance gains over standard RL methods.

推理控制强化学习潜变量建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。