用大模型生成老旧代码文档，发现效果好但评估工具不足
Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation
- 设计提示策略生成逐行代码注释，用多维度评分体系评估质量
- 大模型对MUMPS和主机组装语言的注释基本无幻觉，完整且有用
- 现有自动化指标无法有效预测注释质量，需改进评估方法
老旧软件系统（如用MUMPS和主机汇编语言编写的电子病历系统）在效率、维护、人员和安全方面面临挑战。尽管大语言模型（LLMs）在现代化改造中展现出潜力，但其对老旧语言的理解能力尚不明确。本文基于两个数据集——一个MUMPS电子健康记录系统和开源的IBM主机组装语言代码（ALC）——研究了利用LLMs生成代码文档的可行性。提出一种用于生成逐行注释的提示策略，并建立评估框架，衡量注释的完整性、可读性、实用性及幻觉程度。研究评估了人工评价与代码复杂度、引用基准等自动化指标之间的相关性。结果表明，LLM生成的MUMPS和ALC注释总体上无幻觉，具备完整性、可读性和实用性，优于部分真实注释；但主机组装语言仍存在挑战。然而，目前的自动化指标与注释质量之间无强相关性，难以准确预测或衡量性能。研究揭示了现有自动评估手段的局限性，强调亟需更优的评估框架来支持老旧系统中大模型生成文档的质量判断。
原文摘要 · Abstract (English)
Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。