用模型自身规律生成高质量对齐数据，效率提升近半。
Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation
- 利用模型内部表征方向筛选自生成指令，避免分布偏差。
- 在解码时双向控制表示，实现响应对精准区分，胜率提升超13%。
- 适合追求高效对齐的LLM研发者，尤其适用于资源受限场景。
大语言模型（LLMs）对齐人类偏好需要高质量的偏好数据集。然而，传统构建方法面临两大挑战：依赖预收集指令易导致与目标模型分布不匹配；采样多个随机响应带来巨大计算开销。本文提出一种范式转变——利用大模型表征空间的内在调节机制，高效构建定制化偏好数据集，命名为Icon²。具体而言，首先提取分层方向向量以编码复杂的人类偏好，再据此筛选自生成指令的内在一致性；解码时采用双向内在控制，引导词元表示，精确生成具有明确对齐差异的响应对。实验表明，该方法在对齐性与效率上均有显著提升：Llama3-8B和Qwen2-7B在AlpacaEval 2.0上平均胜率提升13.89%，在Arena-Hard上提升13.45%，计算成本降低最高达48.1%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) require high quality preference datasets to align with human preferences. However, conventional methods for constructing such datasets face significant challenges: reliance on pre-collected instructions often leads to distribution mismatches with target models, while the need for sampling multiple stochastic responses introduces substantial computational overhead. In this work, we explore a paradigm shift by leveraging inherent regulation of LLMs' representation space for efficient and tailored preference dataset construction, named Icon$^{2}$. Specifically, it first extracts layer-wise direction vectors to encode sophisticated human preferences and then uses these vectors to filter self-synthesized instructions based on their inherent consistency. During decoding, bidirectional inherent control is applied to steer token representations, enabling the precise generation of response pairs with clear alignment distinctions. Experimental results demonstrate significant improvements in both alignment and efficiency. Llama3-8B and Qwen2-7B achieve an average win rate improvement of 13.89% on AlpacaEval 2.0 and 13.45% on Arena-Hard, while reducing computational costs by up to 48.1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。