用双层优化让模型从数据中自动学出奖励信号，提升生成效果。
From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation
- 把奖励函数设为外层变量，用策略梯度做内层优化
- 理论分析证明方法可行，实测在分类和强化学习中有效
- 适合只有数据没有奖励信号的场景，如隐私保护训练
生成模型是现代机器学习的核心，支撑着文本、视觉和多模态应用的前沿系统。尽管最大似然估计长期作为主流训练范式，但近期研究指出其在泛化能力与灾难性遗忘方面存在局限，相较强化学习中的策略梯度方法表现更差。然而，这些方法依赖显式奖励信号，而现实中常不可得，导致仅拥有高质量数据时如何对齐生成模型成为根本难题。本文提出一种双层优化框架：将奖励函数视为外层优化变量，以策略梯度目标定义内层问题。我们在可处理设定下开展理论分析，并提炼出可推广至表格分类与基于模型的强化学习等应用的洞见。代码已开源：https://github.com/abenechehab/nll_to_po。
原文摘要 · Abstract (English)
Generative models form the backbone of modern machine learning, underpinning state-of-the-art systems in text, vision, and multimodal applications. While Maximum Likelihood Estimation has traditionally served as the dominant training paradigm, recent work have highlighted its limitations, particularly in generalization and susceptibility to catastrophic forgetting compared to Reinforcement Learning techniques, such as Policy Gradient methods. However, these approaches depend on explicit reward signals, which are often unavailable in practice, leaving open the fundamental problem of how to align generative models when only high-quality datasets are accessible. In this work, we address this challenge via a Bilevel Optimization framework, where the reward function is treated as the optimization variable of an outer-level problem, while a policy gradient objective defines the inner-level. We then conduct a theoretical analysis of this optimization problem in a tractable setting and extract insights that, as we demonstrate, generalize to applications such as tabular classification and model-based reinforcement learning. We release the code at https://github.com/abenechehab/nll_to_po .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。