用多智能体LLM系统自动评估软件架构作业，生成精准反馈。
CAPRA: Scaling Feedback on Software Architecture Deliverables with a Multi-Agent LLM System
- 分角色智能体协同分析文档与图表，确保反馈专业可靠。
- 88.8%评估指标达标，报告处理时间约4分钟，一致性达中等水平。
- 适合教育领域自动化评阅，尤其需结构化反馈的课程设计。
自动化软件工程教育评估在代码评分和论文打分方面已取得显著进展，但对需分析结构完整性和需求可追溯性的软件架构交付物，尚未实现全面自动化。将大语言模型（LLMs）应用于该任务需具备稳健架构以保证技术反馈的准确性与可靠性。本文提出CAPRA（可配置架构能力报告评估）——一种多智能体LLM系统，用于分析软件架构交付物并生成个性化、符合模板的LaTeX反馈。核心设计包括协调多个专用智能体，并采用基于Python的微服务进行多模态文档提取，利用PyMuPDF和视觉增强型LLM（gpt-4o）解析文本与UML图。为保障教育可靠性并减少幻觉，引入基于归一化Levenshtein距离的模糊匹配证据锚定机制，以及一致性管理器智能体进行交叉验证、去重与合并。系统性能通过涵盖五类八项标准的二元评估体系检验：(i)提取完整性，(ii)特征验证，(iii)问题定位与严重性检测，(iv)建议具体性与可追溯性，(v)模板与语气合规性。初步实证评估在10份学生报告上显示，CAPRA在严格双评估员聚合规则下满足88.8%的评价标准，与人类评估者达成中等水平的组间一致性（kappa = 0.582），每份报告处理时间略超4分钟。结果支持了基于LLM的架构反馈可行性，但主观维度仍需人工监督。
原文摘要 · Abstract (English)
Automated assessment in software engineering education has advanced significantly for code grading and essay scoring. However, reviewing software architecture deliverables, which requires analyzing structural completeness and requirements traceability, has not yet been fully automated. Applying Large Language Models (LLMs) to this task requires robust architectures to ensure technical feedback is accurate and reliable for students. This paper presents CAPRA (Configurable Architecture Proficiency Report Assessment), a multi-agent LLM system that analyzes software architecture deliverables to generate personalized, template-compliant LaTeX feedback. As a core design choice, CAPRA coordinates multiple specialized agents and employs a Python-based microservice for multi-modal document extraction, utilizing PyMuPDF and vision-enabled LLMs (specifically gpt-4o) to parse text and UML diagrams. To ensure educational reliability and mitigate hallucinations, CAPRA introduces a deterministic Evidence Anchoring step using fuzzy matching via normalized Levenshtein distance, along with a ConsistencyManager agent that cross-verifies, deduplicates, and merges findings. System performance is assessed using a structured eight-criterion binary evaluation taxonomy covering: (i) extraction completeness, (ii) feature validation, (iii) issue grounding and severity detection, (iv) recommendation specificity and traceability, and (v) template and tone compliance. A preliminary empirical evaluation on 10 student reports shows that CAPRA satisfied 88.8% of the evaluated criteria under a strict two-rater aggregation rule, achieved moderate inter-rater agreement with human evaluators (kappa = 0.582), and processed each report in slightly over 4 minutes. While these results support the viability of LLM-supported architectural feedback, human oversight remains essential for subjective assessment dimensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。