用AI代理自动评估开源软件易用性，省时省力。
OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
- 用大模型驱动的智能代理模拟不同水平开发者编程
- 自动生成代码并多维度评估合规性、正确性和可读性
- 适合开源项目团队快速验证产品易用性
可用性评估对开源软件（OSS）的影响力和采纳至关重要，但传统依赖人工的方法成本高且难以扩展。为此，我们提出OSS-UAgent——一种面向开源软件的自动化、可配置、交互式智能体评估框架。该框架利用大语言模型（LLMs）驱动的智能体，模拟从初级到专家不同经验水平开发者的编程行为。通过动态构建平台特定知识库，确保代码生成的准确性和上下文相关性。生成的代码被自动评估多个维度，包括合规性、正确性和可读性，提供全面的可用性度量。此外，我们的演示展示了OSS-UAgent在图分析平台评估中的实际应用，证明其在自动化可用性评估方面的有效性。
原文摘要 · Abstract (English)
Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。