arXiv:2509.13773cs.AIcs.IR2025-09ACL被引 2

长按图文即可一键调用AI服务,智能推荐精准指令。

MIRA: Empowering One-Touch AI Services on Smartphones with MLLM-based Instruction Recommendation

  • 用多模态大模型分析上下文,理解用户意图并生成指令
  • 通过模板增强推理,提升任务推荐准确率
  • 采用前缀树约束输出,确保建议符合预设指令集

生成式AI的快速发展推动了各类AI服务在智能手机中的集成,改变了用户与设备的交互方式。本文提出MIRA,一种基于多模态大语言模型(MLLM)的任务指令推荐框架,支持用户通过长按图片或文本对象实现一键式AI操作。MIRA引入三项创新:1)基于MLLM的结构化推理推荐流程,可提取关键实体、推断用户意图并生成精确指令;2)融合高层推理模板的增强型推理机制,提升任务推断准确性;3)基于前缀树的约束解码策略,将输出限制在预定义指令候选集中,确保建议连贯且意图对齐。通过真实世界标注数据集评估和用户研究,MIRA在指令推荐准确率上表现显著提升。结果表明,该框架有望彻底改变用户在手机上使用AI服务的方式,带来更流畅高效的体验。

原文摘要 · Abstract (English)

The rapid advancement of generative AI technologies is driving the integration of diverse AI-powered services into smartphones, transforming how users interact with their devices. To simplify access to predefined AI services, this paper introduces MIRA, a pioneering framework for task instruction recommendation that enables intuitive one-touch AI tasking on smartphones. With MIRA, users can long-press on images or text objects to receive contextually relevant instruction recommendations for executing AI tasks. Our work introduces three key innovations: 1) A multimodal large language model (MLLM)-based recommendation pipeline with structured reasoning to extract key entities, infer user intent, and generate precise instructions; 2) A template-augmented reasoning mechanism that integrates high-level reasoning templates, enhancing task inference accuracy; 3) A prefix-tree-based constrained decoding strategy that restricts outputs to predefined instruction candidates, ensuring coherent and intent-aligned suggestions. Through evaluation using a real-world annotated datasets and a user study, MIRA has demonstrated substantial improvements in the accuracy of instruction recommendation. The encouraging results highlight MIRA's potential to revolutionize the way users engage with AI services on their smartphones, offering a more seamless and efficient experience.

手机AI指令推荐多模态大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。