用大模型和多智能体自动评估手机助手的多模态表现。
An Automated Multi-modal Evaluation Framework for Mobile Intelligent Assistants Based on Large Language Models and Multi-Agent Collaboration
- 构建三类智能体协同评估交互、语义和体验。
- 在8个主流助手上准确预测用户满意度，接近人工水平。
- 适合做智能助手研发与评测的团队使用。
随着移动智能助手技术的快速发展，多模态AI助手已成为日常人机交互的核心接口。然而，现有评估方法存在人工成本高、标准不一、主观偏差等问题。本文提出一种基于大语言模型与多智能体协作的自动化多模态评估框架。该框架采用三层智能体架构：交互评估智能体、语义验证智能体和体验决策智能体。通过在Qwen3-8B模型上进行监督微调，实现了与人工专家高度一致的评估匹配度。在八个主流智能助手上的实验表明，该框架能有效预测用户满意度并识别生成缺陷。
原文摘要 · Abstract (English)
With the rapid development of mobile intelligent assistant technologies, multi-modal AI assistants have become essential interfaces for daily user interactions. However, current evaluation methods face challenges including high manual costs, inconsistent standards, and subjective bias. This paper proposes an automated multi-modal evaluation framework based on large language models and multi-agent collaboration. The framework employs a three-tier agent architecture consisting of interaction evaluation agents, semantic verification agents, and experience decision agents. Through supervised fine-tuning on the Qwen3-8B model, we achieve a significant evaluation matching accuracy with human experts. Experimental results on eight major intelligent agents demonstrate the framework's effectiveness in predicting users' satisfaction and identifying generation defects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。