用小模型模仿大模型，实现低成本实时图像风格编辑。
Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications
- 用大模型指导小模型学习工具调用，提升效率
- 小模型性能接近大模型，延迟降低超70%
- 数据少时用增强技术提升25%训练效果
我们提出一种实用的蒸馏方法，用于微调小型语言模型（LLM）以在实时应用中调用工具。聚焦于视觉编辑任务，即通过解析用户自然语言风格请求（如“黄金时刻”），利用LLM选择合适的工具及其参数来实现期望的视觉效果。我们发现，尽管专有大模型（如GPT-3.5-Turbo）在此任务中表现良好，但其高成本和延迟使其难以用于实时场景。为此，我们采用教师-学生框架，用大型教师模型与行为信号指导小型学生模型的微调，并引入离线评估指标。在线与离线实验表明，学生模型可达到与教师模型（GPT-3.5-Turbo）相当的性能，显著降低计算成本与响应延迟。此外，在低数据场景下，通过数据增强使微调效果提升25%。
原文摘要 · Abstract (English)
We present a practical distillation approach to fine-tune LLMs for invoking tools in real-time applications. We focus on visual editing tasks; specifically, we modify images and videos by interpreting user stylistic requests, specified in natural language ("golden hour"), using an LLM to select the appropriate tools and their parameters to achieve the desired visual effect. We found that proprietary LLMs such as GPT-3.5-Turbo show potential in this task, but their high cost and latency make them unsuitable for real-time applications. In our approach, we fine-tune a (smaller) student LLM with guidance from a (larger) teacher LLM and behavioral signals. We introduce offline metrics to evaluate student LLMs. Both online and offline experiments show that our student models manage to match the performance of our teacher model (GPT-3.5-Turbo), significantly reducing costs and latency. Lastly, we show that fine-tuning was improved by 25% in low-data regimes using augmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。