让模型通过自我一致性优化，无需人工标注就能提升复杂推理能力。
Self-Consistency Preference Optimization
- 用自洽性原则迭代训练模型，偏好一致答案、排斥不一致答案。
- 在GSM8K和MATH上超越传统奖励模型训练，接近有监督最优水平。
- 可使小模型性能超越大模型，适合资源有限但需强推理的场景。
自我对齐技术使模型能在无人工标注的情况下自我改进，但现有方法在复杂推理任务上表现不佳，主要因难以正确分配奖励。一种已知能提高准确性的方法是推理时通过多次采样实现自洽性，以寻找最一致的答案。本文将这一思想扩展到训练阶段，提出自洽性偏好优化（ScPO），通过迭代训练使一致答案优于不一致答案，且在无监督新问题上进行。实验表明，ScPO在GSM8K和MATH等推理任务上显著优于传统奖励模型训练，接近使用真实答案或偏好进行的监督训练效果；将ScPO与标准监督学习结合，进一步提升性能。在ZebraLogic测试中，使用ScPO微调后的Llama-3 8B模型表现优于Llama-3 70B、Gemma-2 27B和Claude-3 Haiku。
原文摘要 · Abstract (English)
Self-alignment, whereby models learn to improve themselves without human annotation, is a rapidly growing research area. However, existing techniques often fail to improve complex reasoning tasks due to the difficulty of assigning correct rewards. An orthogonal approach that is known to improve correctness is self-consistency, a method applied at inference time based on multiple sampling in order to find the most consistent answer. In this work, we extend the self-consistency concept to help train models. We thus introduce self-consistency preference optimization (ScPO), which iteratively trains consistent answers to be preferred over inconsistent ones on unsupervised new problems. We show ScPO leads to large improvements over conventional reward model training on reasoning tasks such as GSM8K and MATH, closing the gap with supervised training with gold answers or preferences, and that combining ScPO with standard supervised learning improves results even further. On ZebraLogic, ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and Claude-3 Haiku.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。