不微调大模型，用注意力机制实现界面元素精准定位。
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
- 通过构造特定提示词提取大模型的注意力图进行定位。
- 在文本定位任务上达到与微调方法相当的效果。
- 适合想快速部署无训练界面理解系统的开发者。
多模态大模型在自主交互和解析图形用户界面（GUI）方面展现出巨大潜力。核心挑战在于准确识别关键界面组件（如文字、图标），这通常依赖于针对特定数据集对大模型进行微调以直接预测组件位置。本文提出无需微调的注意力驱动接地（TAG）方法，利用预训练多模态大模型固有的注意力模式完成此任务。该方法通过精心设计的查询提示，识别并聚合特定标记的注意力图。应用于当前领先的多模态大模型MiniCPM-Llama3-V 2.5，其无微调方案在性能上可媲美基于微调的方法，尤其在文本定位任务中表现优异。此外，我们证明了基于注意力图的定位方式显著优于模型直接输出的位置预测，凸显了预训练大模型注意力信息的潜力，为该领域未来发展开辟新路径。
原文摘要 · Abstract (English)
Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding-accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。