让生成模型同时满足多种人类偏好，且不牺牲任何一项表现。
MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models
- 并行训练多个偏好专用的LoRA专家，再逐步合并优化基础模型。
- 在多模态任务中，各项评估指标最高提升67.1%，显著超越现有方法。
- 适合需要兼顾美观、实用与安全性的生成系统研发人员。
基于人类反馈的强化学习（RLHF）通过奖励模型推动生成模型对人类审美与感知偏好的对齐。然而，联合优化多个奖励常导致对齐代价，即提升某一维度时损害其他维度。为此，本文提出两种互补方法：MapReduce LoRA 和奖励感知词嵌入（RaTE）。MapReduce LoRA 并行训练特定偏好的 LoRA 专家，并迭代合并以精炼共享基础模型；RaTE 学习奖励相关的词嵌入，在推理时动态组合，实现灵活偏好控制。在文本到图像生成任务（Stable Diffusion 3.5 Medium 和 FLUX.1-dev）上，GenEval、PickScore 与 OCR 指标分别提升 36.1%、4.6%、55.7% 和 32.7%、4.3%、67.1%。在文本到视频生成（HunyuanVideo）中，视觉质量和运动质量分别提升 48.1% 和 90.0%。在语言任务中，使用 Llama-2 7B 的 Helpful Assistant，有用性和无害性分别提升 43.4% 和 136.7%。该框架在跨模态多偏好对齐上达到新最优水平。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax, improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。