用对抗式模仿学习微调流模型，无需奖励或成对比较。
FAIL: Flow Matching Adversarial Imitation Learning for Image Generation
- 通过对抗训练最小化策略与专家差异，避免依赖奖励信号。
- 仅用1.3万条演示数据，微调FLUX在提示跟随和美学评分上表现优异。
- 适用于图像、视频生成，还能防止奖励滥用，适合多场景应用。
流模型的后训练过程在数学上等价于模仿学习。虽然监督微调能有效模仿专家示范，但无法纠正未见状态下的策略漂移。偏好优化方法虽可解决此问题，却需代价高昂的偏好对或奖励建模。本文提出流匹配对抗模仿学习（FAIL），通过对抗训练最小化策略与专家分布差异，无需显式奖励或成对比较。我们推导出两种算法：FAIL-PD利用可微分常微分方程求解器获得低方差路径梯度；FAIL-PG则提供一种适用于离散或计算受限场景的黑箱替代方案。仅使用13,000条来自Nano Banana pro的演示数据对FLUX进行微调，即可在提示跟随与美学基准测试中达到竞争力表现。此外，该框架在离散图像与视频生成中具有良好泛化能力，并可作为鲁棒正则化项，缓解基于奖励优化中的奖励黑客问题。代码与数据见https://github.com/HansPolo113/FAIL。
原文摘要 · Abstract (English)
Post-training of flow matching models-aligning the output distribution with a high-quality target-is mathematically equivalent to imitation learning. While Supervised Fine-Tuning mimics expert demonstrations effectively, it cannot correct policy drift in unseen states. Preference optimization methods address this but require costly preference pairs or reward modeling. We propose Flow Matching Adversarial Imitation Learning (FAIL), which minimizes policy-expert divergence through adversarial training without explicit rewards or pairwise comparisons. We derive two algorithms: FAIL-PD exploits differentiable ODE solvers for low-variance pathwise gradients, while FAIL-PG provides a black-box alternative for discrete or computationally constrained settings. Fine-tuning FLUX with only 13,000 demonstrations from Nano Banana pro, FAIL achieves competitive performance on prompt following and aesthetic benchmarks. Furthermore, the framework generalizes effectively to discrete image and video generation, and functions as a robust regularizer to mitigate reward hacking in reward-based optimization. Code and data are available at https://github.com/HansPolo113/FAIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。