评估基于COBOL代码的生成系统对输入扰动的鲁棒性，助力修复敏感问题。
Evaluating perturbation robustness of generative systems that use COBOL code inputs
- 构建COBOL段落与程序级扰动方法库，生成变体数据集。
- 通过输出指标变化量化系统对输入微小改动的敏感度。
- 提供动态可视化工具，帮助定位并改进系统脆弱点。
将大语言模型（LLM）作为组件的系统常对不改变语义的输入微小变化表现出敏感性（即非鲁棒性），影响实际可用性。本文提出一个针对以COBOL代码为输入的生成系统鲁棒性的评估框架，应用场景为COBOL与Java之间的代码转换，但可扩展至代码生成或解释等任务。由于大量关键业务系统使用COBOL编写，而这些遗留系统多为专有且无法用于训练LLM，因此评估其鲁棒性尤为必要且具挑战性。我们开发了针对COBOL段落和完整程序的扰动方法库，并创建了特定任务的基准数据集变体扩展版本。通过测量系统输出中各项指标及聚合指标的变化，评估LLM系统在输入扰动下的表现。最后，我们设计了一系列动态表格与图表可视化仪表盘，辅助调试系统输出，监控并理解系统对输入变化的敏感根源。这些工具还可用于优化系统，例如识别应由预处理步骤处理的输入变异类型。
原文摘要 · Abstract (English)
Systems incorporating large language models (LLMs) as a component are known to be sensitive (i.e., non-robust) to minor input variations that do not change the meaning of the input; such sensitivity may reduce the system's usefulness. Here, we present a framework to evaluate robustness of systems using COBOL code as input; our application is translation between COBOL and Java programming languages, but the approach extends to other tasks such as code generation or explanation. Targeting robustness of systems with COBOL as input is essential yet challenging. Many business-critical applications are written in COBOL, yet these are typically proprietary legacy applications and their code is unavailable to LLMs for training. We develop a library of COBOL paragraph and full-program perturbation methods, and create variant-expanded versions of a benchmark dataset of examples for a specific task. The robustness of the LLM-based system is evaluated by measuring changes in values of individual and aggregate metrics calculated on the system's outputs. Finally, we present a series of dynamic table and chart visualization dashboards that assist in debugging the system's outputs, and monitoring and understanding root causes of the system's sensitivity to input variation. These tools can be further used to improve the system by, for instance, indicating variations that should be handled by pre-processing steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。