arXiv:2608.06347cs.CL2026-08

通过推理枢纽引导,提升多语言模型的推理迁移能力

RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

论文配图:RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
图 1 · 摘自论文原文
  • 以推理枢纽为指导,聚焦关键决策步骤进行自蒸馏
  • 在17种语言的数学推理任务中超越现有基线方法
  • 适合需要跨语言推理能力增强的研究与应用

多语言推理迁移对扩展大语言模型在低资源语言中的推理能力至关重要。基于策略的自蒸馏(OPSD)及其变体虽能提供密集的标记级监督,但其目标未明确关注跨语言迁移中最关键的推理信号。我们指出,目标语言的推理包含表面文本生成和推理枢纽生成,后者是推动或调整推理过程并影响后续推断的关键决策。因此,我们提出RP-OPSD:推理枢纽引导的在策略自蒸馏,利用带有与不带英文参考解的教师视图之间的分布偏移作为操作代理,引导特权蒸馏与参考锚定。在涵盖17种语言和多个难度级别的数学推理基准上实验表明,该方法优于强基线及OPSD变体。进一步分析显示,RP-OPSD将特权蒸馏集中在推理控制与问题条件状态更新的标记上,而降低对主要支持表面实现的标记的关注。代码已开源。

原文摘要 · Abstract (English)

Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.

多语言推理自蒸馏推理迁移大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。