利用模型内部隐状态几何结构,无须人工标注即可提升大模型推理能力。
Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning

- 基于隐藏状态点云的拓扑结构识别正确与错误推理路径
- 在GSM8K和MATH-Numeric上提升4.49%和4.19%准确率
- 适合需要低成本标注的半监督推理优化场景
偏好优化可提升大语言模型在数学推理上的表现,但可靠的优选-劣选对通常依赖验证答案、人工标注或外部奖励模型。本文探讨是否可在半监督设置下,从模型内部表示的几何结构中提取偏好监督信号。分析表明,不同数学问题生成的推理轨迹在全局空间中形成具有结构的点云,正确与错误轨迹呈现不同的几何组织。据此提出Cloud-ScPO框架,利用少量标注数据构建多个正确与错误参考点云。每个轨迹由均值池化的隐藏状态表示,并通过跨参考库的组件级软k近邻度量进行评分。该点云信号与提示级别的自一致性结合:自一致性确定答案层级偏好方向,而点云评分则筛选具体轨迹并按得分差值过滤样本对。在GSM8K和MATH-Numeric四个模型设置下的实验显示,Cloud-ScPO持续优于ScPO,分别提升4.49%和4.19%。配对级分析进一步表明,Cloud-ScPO在保持正确性可靠性的同时,更有效区分高质量优选轨迹与不完整、重复或低质量的劣选响应。
原文摘要 · Abstract (English)
Preference optimization improves mathematical reasoning in large language models (LLMs), but reliable chosen-rejected pairs usually require verified answers, human annotations, or external reward models. We investigate whether preference supervision can instead be derived from the model's internal representation geometry in a semi-supervised setting. Our analysis shows that reasoning trajectories generated across different mathematical problems form structured global point clouds in which correct and incorrect trajectories exhibit different geometric organization. Based on this observation, we propose Cloud-ScPO, a topology-guided preference-mining framework that uses a small labeled set to construct multiple correct and incorrect reference Clouds. Each trajectory is represented by a mean-pooled hidden state and scored against connectivity-induced components using a component-level soft $k$-nearest-neighbor measure averaged across reference banks. We combine this cross-problem Cloud signal with prompt-level self-consistency: self-consistency determines the answer-level preference direction, while Cloud scoring selects concrete trajectories and filters pairs by their score margin. Experiments on GSM8K and MATH-Numeric across four model settings show that Cloud-ScPO consistently improves over ScPO, with gains of up to 4.49% on GSM8K and 4.19% on MATH-Numeric. Pair-level analyses further show that Cloud-ScPO maintains comparable correctness reliability while more effectively separating informative chosen trajectories from incomplete, repetitive, or otherwise low-quality rejected responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。