提出流模型的连续轨迹监督方法，提升生成质量与收敛速度。
FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models
- 基于同一状态出发的轨迹匹配，设计时序加权速度匹配目标。
- 在多参考框架下，使生成评估、OCR等指标分别提升0.03、0.02、0.31。
- 适用于需要高精度轨迹生成的场景，尤其适合强化学习后训练任务。
尽管在语言模型后训练中，基于策略的蒸馏（OPD）能有效缓解稀疏奖励与暴露偏差问题，但其在流模型中的应用仍不充分。为此，本文提出连续轨迹监督（FlowCTS），通过匹配从相同学生模型访问状态出发的后续学生与参考轨迹来实现。利用轨迹与速度场之间的积分关系，推导出时序加权的速度匹配上界，并离散化为依赖监督步数的可实践目标。在多参考设置下，单状态的FlowCTS-OPD相比传统基于KL的OPD收敛更快，性能更优：生成评估从0.90提升至0.93，OCR从0.90提升至0.92，PickScore从22.75提升至23.06，且优于混合奖励强化学习基线。进一步分析揭示，传统基于KL的OPD因辅助SDE转移核存在明显时序监督失配。在非基于策略设置下，FlowCTS亦持续优于标准监督微调（SFT），尤其在OCR任务上表现突出；增加监督步数虽带来更丰富的轨迹信息，但优化难度随之上升。
原文摘要 · Abstract (English)
While on-policy distillation (OPD) effectively addresses sparse rewards and exposure bias in large language model post-training, its extension to flow models remains underexplored. To this end, we propose Flow Continuous Trajectory Supervision (FlowCTS), which matches subsequent student and reference trajectories initialized from the same student-visited state. Using the integral relation between trajectories and velocity fields, we derive a temporally weighted velocity-matching upper bound and discretize it into practical objectives parameterized by the number of supervision steps. Under a multi-reference setup, single-state FlowCTS-OPD outperforms vanilla KL-based OPD with faster convergence. FlowCTS-OPD improves GenEval from 0.90 to 0.93, OCR from 0.90 to 0.92, and PickScore from 22.75 to 23.06, while outperforming a mixed-reward RL baseline across all target metrics. Further analysis reveals a clear temporal supervision mismatch in vanilla KL-based OPD arising from its auxiliary SDE transition kernels. Beyond on-policy setting,FlowCTS also consistently outperforms vanilla SFT , particularly on OCR, while increasing supervision steps exhibit a trade-off between richer trajectory information and greater optimization difficulty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。