用离线模拟+LLM生成可执行脚本,让非程序员也能自动操作软件。
Skill Discovery for Software Scripting Automation via Offline Simulations with LLMs
- 通过上下文引导和API关联分析生成任务,自动发现可用脚本。
- 在Adobe Illustrator上实测成功率提升,响应时间与调用成本大幅降低。
- 适合想自动化专业软件但不会编程的用户,或研究AI与领域工具结合的开发者。
脚本接口能帮助用户自动化任务并定制工作流,但传统方式需编程技能和对特定API的熟悉,对多数用户构成障碍。尽管大语言模型(LLMs)可从自然语言生成代码,但运行时生成受限于代码未经验证、安全风险高、响应慢及计算成本高。为此,我们提出一种离线仿真框架,利用LLMs和公开脚本指南构建经验证的软件专属技能集。该框架包含两部分:(1) 任务生成,结合自上而下的功能指导与自下而上的API协同探索;(2) 基于执行反馈的脚本生成与优化。为高效探索广泛API,引入基于图神经网络(GNN)的链接预测模型,捕捉API协同关系,生成涉及低使用率API的技能,提升技能多样性。在Adobe Illustrator上的实验表明,该框架显著提高自动化成功率,降低响应时间与运行时令牌消耗。这是首次将软件脚本接口作为LLM系统测试平台,凸显了在受控环境中利用执行反馈的优势,并为对齐人工智能能力与用户需求提供了关键洞见。
原文摘要 · Abstract (English)
Scripting interfaces enable users to automate tasks and customize software workflows, but creating scripts traditionally requires programming expertise and familiarity with specific APIs, posing barriers for many users. While Large Language Models (LLMs) can generate code from natural language queries, runtime code generation is severely limited due to unverified code, security risks, longer response times, and higher computational costs. To bridge the gap, we propose an offline simulation framework to curate a software-specific skillset, a collection of verified scripts, by exploiting LLMs and publicly available scripting guides. Our framework comprises two components: (1) task creation, using top-down functionality guidance and bottom-up API synergy exploration to generate helpful tasks; and (2) skill generation with trials, refining and validating scripts based on execution feedback. To efficiently navigate the extensive API landscape, we introduce a Graph Neural Network (GNN)-based link prediction model to capture API synergy, enabling the generation of skills involving underutilized APIs and expanding the skillset's diversity. Experiments with Adobe Illustrator demonstrate that our framework significantly improves automation success rates, reduces response time, and saves runtime token costs compared to traditional runtime code generation. This is the first attempt to use software scripting interfaces as a testbed for LLM-based systems, highlighting the advantages of leveraging execution feedback in a controlled environment and offering valuable insights into aligning AI capabilities with user needs in specialized software domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。