用隐空间生成偏好数据，18倍快且效果更好
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
- 在模型隐空间做可控扰动生成合成偏好对
- 比文本生成快18倍，用16000倍小的模型达到更优效果
- 适合数据稀缺时提升奖励模型性能
奖励建模对于对齐大语言模型与人类偏好至关重要，但常受限于偏好数据的高昂成本。现有文本数据合成方法计算开销大。本文提出一种新框架LENS，直接在大语言模型的隐空间中合成偏好数据。该方法使用变分自编码器（VAE）学习响应嵌入的结构化隐表示，通过在隐空间进行受控扰动并解码回嵌入空间，高效生成语义一致、多样化的合成偏好对，避免了昂贵的文本生成与标注。我们提供了理论保证：合成对近似保持原始偏好顺序，并提升奖励模型泛化能力。实验表明，隐空间合成在标准基准上显著优于基于文本的增强方法，生成速度提升18倍，所用模型规模仅为原模型的1/16000。本工作为奖励建模提供了高效可扩展的数据增强方案。代码已公开于https://github.com/deeplearning-wisc/lens。
原文摘要 · Abstract (English)
Reward modeling, crucial for aligning large language models (LLMs) with human preferences, is often bottlenecked by the high cost of preference data. Existing textual data synthesis methods are computationally expensive. We propose a novel framework LENS for synthesizing preference data directly in the LLM's latent embedding space. Our method employs a Variational Autoencoder (VAE) to learn a structured latent representation of response embeddings. By performing controlled perturbations in this latent space and decoding back to the embedding space, we efficiently generate diverse, semantically consistent synthetic preference pairs, bypassing costly text generation and annotation. We provide theoretical guarantees that our synthesized pairs approximately preserve original preference ordering and improve reward model generalization. Empirically, our latent-space synthesis significantly outperforms text-based augmentation on standard benchmarks, achieving superior results while being 18x faster in generation and using a 16,000x smaller model. Our work offers a scalable and effective alternative for enhancing reward modeling through efficient data augmentation. Code is publicly available at https://github.com/deeplearning-wisc/lens
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。