无需训练和人工标注，模型自动生成最优对齐指令
Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
- 通过动态奖励机制迭代优化提示词，实现无调优自对齐
- 8个主流大模型均表现超越微调版本，最佳提示优于专家设计
- 适合追求低成本对齐的开发者与研究者
传统大语言模型对齐依赖昂贵训练和人工偏好标注。自对齐旨在降低此类成本。本文提出一种无需调优的自对齐方法——动态奖励提示优化（DRPO），利用基于搜索的优化框架，使大模型在不增加训练或人工干预的前提下，自主迭代改进并生成最优对齐提示。其核心是动态奖励机制，能识别并修正模型特定的对齐弱点，高效适应多样对齐挑战。在8个近期大模型（含开源与闭源）上的实证评估显示，未经微调的基础模型性能已超过经SFT/RLHF微调的版本；且由DRPO自动优化的提示词优于人类专家设计，验证了该方法的有效性。结果表明，当前大模型可通过推理时优化实现自适应对齐，为调优式对齐提供有力补充。
原文摘要 · Abstract (English)
Aligning Large Language Models (LLMs) traditionally relies on costly training and human preference annotations. Self-alignment seeks to reduce these expenses by enabling models to align themselves. To further lower costs and achieve alignment without any expensive tuning or annotations, we introduce a new tuning-free approach for self-alignment, Dynamic Rewarding with Prompt Optimization (DRPO). Our approach leverages a search-based optimization framework that allows LLMs to iteratively self-improve and craft the optimal alignment instructions, all without additional training or human intervention. The core of DRPO is a dynamic rewarding mechanism, which identifies and rectifies model-specific alignment weaknesses, allowing LLMs to adapt efficiently to diverse alignment challenges. Empirical evaluations on eight recent LLMs, both open- and closed-sourced, demonstrate that DRPO significantly enhances alignment performance, with base models outperforming their SFT/RLHF-tuned counterparts. Moreover, the prompts automatically optimized by DRPO surpass those curated by human experts, further validating the effectiveness of our approach. Our findings highlight the great potential of current LLMs to achieve adaptive self-alignment through inference-time optimization, complementing tuning-based alignment methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。