为工业管理设计的智能界面代理框架，提升自动化可靠性与安全性。
InfraMind: A Novel Exploration-based GUI Agentic Framework for Mission-critical Industrial Management
- 基于探索式虚拟机快照理解复杂界面元素
- 在真实工业平台中任务成功率超现有方法，效率显著提升
- 适合需要高安全性的数据中心等关键设施运维人员
关键工业基础设施（如数据中心）依赖复杂的管理软件，但系统复杂度上升、多厂商集成及专家短缺带来巨大挑战。传统RPA脚本灵活性差、维护成本高；现有大模型驱动的GUI代理在工业场景中仍面临五大难题：元素理解不熟、精度效率不足、状态定位困难、部署受限、安全要求难满足。为此，我们提出InfraMind——一种专为工业管理设计的探索式GUI智能体框架。该框架集成五项创新模块：（1）基于虚拟机快照的系统性搜索探索，实现对复杂界面的自主理解；（2）记忆驱动规划，保障高精度高效执行；（3）先进状态识别，增强层级界面中的定位鲁棒性；（4）结构化知识蒸馏，支持轻量模型高效部署；（5）多层次综合安全机制，保护敏感操作。在开源与商业DCIM平台上的大量实验表明，该方法在任务成功率和运行效率上均优于现有框架，提供了一种严谨且可扩展的工业自动化解决方案。
原文摘要 · Abstract (English)
Mission-critical industrial infrastructure, such as data centers, increasingly depends on complex management software. Its operations, however, pose significant challenges due to the escalating system complexity, multi-vendor integration, and a shortage of expert operators. While Robotic Process Automation (RPA) offers partial automation through handcrafted scripts, it suffers from limited flexibility and high maintenance costs. Recent advances in Large Language Model (LLM)-based graphical user interface (GUI) agents have enabled more flexible automation, yet these general-purpose agents face five critical challenges when applied to industrial management, including unfamiliar element understanding, precision and efficiency, state localization, deployment constraints, and safety requirements. To address these issues, we propose InfraMind, a novel exploration-based GUI agentic framework specifically tailored for industrial management systems. InfraMind integrates five innovative modules to systematically resolve different challenges in industrial management: (1) systematic search-based exploration with virtual machine snapshots for autonomous understanding of complex GUIs; (2) memory-driven planning to ensure high-precision and efficient task execution; (3) advanced state identification for robust localization in hierarchical interfaces; (4) structured knowledge distillation for efficient deployment with lightweight models; and (5) comprehensive, multi-layered safety mechanisms to safeguard sensitive operations. Extensive experiments on both open-source and commercial DCIM platforms demonstrate that our approach consistently outperforms existing frameworks in terms of task success rate and operational efficiency, providing a rigorous and scalable solution for industrial management automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。