提出新方法ZOPrO，用零阶优化提升大模型生成质量。
Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models
- 基于策略与奖励模型互动规律，改进随机逼近算法
- 在摘要、翻译等任务上收敛速度接近一阶方法
- 首次将零阶优化用于大模型偏好学习，适合资源受限场景
使用梯度的微调方法计算开销大。零阶（ZO）优化仅通过函数值评估降低内存占用，但高维模型中收敛慢。现有研究多集中于分类任务，忽略生成类任务。本文提出ZOPrO，一种面向大语言模型偏好优化的新型零阶算法。我们分析了传统一阶偏好优化中策略与奖励模型的更新交互模式，据此改进同时扰动随机逼近（SPSA），引入定向采样策略加速收敛。在摘要生成、机器翻译和对话助手任务上的实验表明，该方法能持续增强奖励信号，且收敛时间接近一阶方法。虽不及部分顶尖方法，但这是首个将零阶优化应用于大模型偏好优化的工作，突破了分类任务限制，开启全新研究方向。代码与可视化见https://github.com/alessioGalatolo/VisZOPrO。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) with first-order methods like back-propagation is computationally intensive. Zeroth-Order (ZO) optimisation uses function evaluations instead of gradients, reducing memory usage, but suffers from slow convergence in high-dimensional models. As a result, ZO research in LLMs has mostly focused on classification, overlooking more complex generative tasks. In this paper, we introduce ZOPrO, a novel ZO algorithm designed for Preference Optimisation in LLMs. We begin by analysing the interplay between policy and reward models during traditional (first-order) Preference Optimisation, uncovering patterns in their relative updates. Guided by these insights, we adapt Simultaneous Perturbation Stochastic Approximation (SPSA) with a targeted sampling strategy to accelerate convergence. Through experiments on summarisation, machine translation, and conversational assistants, we demonstrate that our method consistently enhances reward signals while achieving convergence times comparable to first-order methods. While it falls short of some state-of-the-art methods, our work is the first to apply Zeroth-Order methods to Preference Optimisation in LLMs, going beyond classification tasks and paving the way for a largely unexplored research direction. Code and visualisations are available at https://github.com/alessioGalatolo/VisZOPrO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。