通过几何旋转引导语言模型推理,显著提升数学、科学和代码任务准确率。
Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning

- 利用正交旋转将输入向量映射到推理空间,实现无需训练的思维引导。
- 在32组测试中30组准确率提升,多任务及跨模态零样本泛化有效。
- 方法简洁高效,适用于各类大模型,尤其适合缺乏标注数据的推理场景。
推理模型通过生成显式的中间推理路径来提升复杂任务表现。然而,模型在推理过程中的隐藏表示结构仍不明确:推理过程中的隐藏表示与输入提示的嵌入有何差异?这种结构能否在推理时被利用以增强推理能力?我们发现,输入嵌入与推理嵌入(对提示和推理轨迹的最后层隐藏状态进行均值池化)均具有极高的锥度性,所有向量紧密聚集于单一均值方向。关键的是,这些均值输入方向与推理方向非共线,在多个模型和基准任务中占据嵌入空间中几何上不同的区域。这一现象启发我们将输入到推理的转换视为一个可通过正交普鲁斯特分析求解的旋转问题。我们提出 Rotate2Think,一种无需训练的方法:从少量正确解答样本中估计该旋转,并在推理开始时在思维分隔符之间注入合成的推理向量,作为初始几何引导。在多个基准测试和模型族上的评估显示,Rotate2Think 在32个模型-基准配置中提升了30个,且在 MATH-Vision 上实现了多模态推理的零样本泛化。
原文摘要 · Abstract (English)
Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer. Yet the internal structure of representation space when reasoning remains poorly understood: how do a model's hidden representations differ during thinking versus the embeddings of the input prompt, and can this structure be exploited to elicit stronger reasoning at inference time? We show that both input embeddings and thinking embeddings (mean-pooled last-layer hidden states over the prompt and reasoning trace, respectively) exhibit extremely high conicity, with all vectors clustering tightly around a single mean direction. Crucially, these mean input and thinking directions are non-collinear, with thinking embeddings occupying a geometrically distinct region of embedding space across many different models and benchmark tasks. This observation motivates casting the input-to-thinking transition as a rotation problem admitting a closed-form solution via orthogonal Procrustes analysis. We propose Rotate2Think, a training-free method that estimates this rotation from a small set of correctly solved examples and injects the resulting synthetic thinking vector between thinking delimiters at inference time, providing a geometric primer at the onset of the reasoning trace. Evaluated across multiple benchmarks and model families, Rotate2Think improves accuracy in 30 of 32 model-benchmark configurations across mathematics, science, and code tasks, and generalizes zero-shot to multimodal reasoning on MATH-Vision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。