用视觉艺术优化冻结模型，实现高效多模态微调。
Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training
- 通过优化输入图像像素实现无需修改计算图的微调。
- 在数学与工具使用任务上性能媲美LoRA。
- 生成的任务相关艺术化输入适合创意应用。
大型语言模型（LLM）主要有两种参数高效微调（PEFT）方法：低秩适配（LoRA）在层间引入额外权重，软提示（Soft Prompting）则在输入中添加特定可训练的原始标记。但两者均需修改预编译、预优化的LLM计算图，无法在高吞吐引擎如vLLM中完全支持。本文提出基于艺术的强化训练（ART），通过仅优化冻结多模态大模型（MLLM）的原始视觉输入来注入信息，从而在预编译计算图上实现软提示方法。该方法依赖梯度反向传播至像素数组，兼容任意微调目标。此外，优化后的视觉输入可被风格化为任务相关的计算艺术作品。实验验证了该方法在多种规模的开源Qwen架构及多个文本基准上的有效性。具体而言，ART在数学和结构化工具使用基准上达到与LoRA相当的准确率。
原文摘要 · Abstract (English)
There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。