让大模型直接操作软件源码和运行时，实现更智能的自动化
Every Software as an Agent: Blueprint and Case Study
- 让大模型直接访问软件源码和运行环境，动态注入代码执行
- 在两个桌面应用上验证了方案可行性，提升指令理解与执行效率
- 适合想构建高阶自动化工具的研究者与开发者
大型语言模型（LLM）的兴起揭示了软件代理的可能性——即软件能理解并执行自然语言指令。然而，现有的基于API或GUI的代理在准确性和效率上仍不理想。本文主张赋予LLM对软件内部结构（源码和运行时上下文）的访问权限，并允许其动态注入生成代码进行执行。在此白盒设定下，可更充分地利用软件上下文与大模型的编码能力。我们提出整体设计架构，并以两款流行的基于Web的桌面应用为例进行了案例研究。同时深入探讨了挑战与未来方向。我们认为这一新范式有望从根本上重构软件代理设计，最终实现一个软件能理解、操作、协作甚至思考以满足复杂用户需求的数字世界。
原文摘要 · Abstract (English)
The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。