用高质量数据和新评估法提升大模型生成Bash命令的准确率
LLM-Supported Natural Language to Bash Translation
- 构建600对人工验证的指令-命令数据集，提升评估可靠性
- 新方法实现95%置信度的功能等价判断，比旧方法高16%
- 多种优化技术使翻译准确率最高提升32%，适合系统开发者
Linux系统的Bash命令行接口语法复杂且需专业知识。利用大语言模型（LLMs）的自然语言转Bash命令（NL2SH）能力可绕过此问题。但现有评估因测试数据不准确、功能等价判断不可靠而困难。本文构建了600对人工验证的指令-命令对测试集和40,939对训练集，分别比之前数据集扩大441%和135%。提出一种结合命令执行与LLM输出评估的新功能等价判别方法，在95%置信度下优于旧方法16%。使用该测试集和方法评估主流LLMs发现，解析、上下文学习、权重内学习和约束解码可使NL2SH准确率最高提升32%。研究强调数据质量、执行式评估和翻译方法对推进NL2SH的重要性。代码已开源。
原文摘要 · Abstract (English)
The Bourne-Again Shell (Bash) command-line interface for Linux systems has complex syntax and requires extensive specialized knowledge. Using the natural language to Bash command (NL2SH) translation capabilities of large language models (LLMs) for command composition circumvents these issues. However, the NL2SH performance of LLMs is difficult to assess due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. We present a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous datasets by 441% and 135%, respectively. Further, we present a novel functional equivalence heuristic that combines command execution with LLM evaluation of command outputs. Our heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heuristics. Evaluation of popular LLMs using our test dataset and heuristic demonstrates that parsing, in-context learning, in-weight learning, and constrained decoding can improve NL2SH accuracy by up to 32%. Our findings emphasize the importance of dataset quality, execution-based evaluation and translation method for advancing NL2SH translation. Our code is available at https://github.com/westenfelder/NL2SH
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。