用可学习网络自动优化多模态大模型的训练数据配比，提升手机助手性能。
DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents
- 设计可训练网络预测不同数据比例下的任务表现，找到最优混合策略。
- 在1235个真实场景问答上提升3.38%，跨基准平均高出2.57%。
- 适用于多种模型架构，尤其适合移动端多任务大模型优化。
手机助手（MPAs）因其广泛适用性成为研究热点，而多模态大语言模型（MLLMs）是其基础。然而现有模型在同时处理多项手机任务时表现有限。尽管多任务监督微调（SFT）被广泛应用，但如何确定最佳训练数据组合仍无有效方法。为此，我们提出DaMo（数据混合优化器），通过一个可训练网络预测任意数据比例下的下游任务性能，从而实现最优混合配置。为支持全面评估，我们构建了首个专门针对多模态手机任务的基准测试PhoneAgentBench，包含1235个涵盖真实工业场景的问答对。小规模实验显示其预测能力优异（R²=0.81），能高效推断最优数据混合。结果表明，与替代方法相比，DaMo在PhoneAgentBench上提升3.38%；在BFCL-v3、MME-Reasoning、MME-Perception和OCRBench等主流基准上，平均分提升2.57%。仅在BFCL-v3任务中，优化效果提升12.47%。此外，DaMo具备良好可扩展性，适用于其他模型架构。代码与数据已开源。
原文摘要 · Abstract (English)
Mobile Phone Agents (MPAs) have emerged as a promising research direction due to their broad applicability across diverse scenarios. While Multimodal Large Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness in handling multiple mobile phone tasks simultaneously remains limited. Although multitask supervised fine-tuning (SFT) is widely adopted for multitask learning, existing approaches struggle to determine optimal training data compositions for peak performance. To address this challenge, we propose DaMo (Data Mixture Optimizer) - a novel solution employing a trainable network that predicts optimal data mixtures by forecasting downstream task performance for any given dataset ratio. To support comprehensive evaluation, we introduce PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse real-world industrial mobile application scenarios. Demonstrating strong predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo efficiently extrapolates optimal data mixing configurations. Our results show DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods. Furthermore, extensive experiments across established benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench reveal DaMo's superior generalization, outperforming other approaches by 2.57% in terms of average score. When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably, DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures. The code and dataset are available at https://github.com/OPPO-Mente-Lab/DaMo.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。