小模型也能写出好代码注释，关键在提示词设计。
NLD-LLM: A systematic framework for evaluating small language transformer models on natural language description
- 用标准化提示词统一评估不同大小的Transformer模型。
- 小模型经优化提示后，性能可媲美大模型。
- 适合关注模型效率与提示工程的研究者。
自然语言描述（NLD）是自然语言处理中的一项任务，要求模型从自然语言输入生成结构化且有意义的输出。本文提出NLD-LLM，一个系统化的NLP评估框架，用于衡量语言模型生成准确、简洁源代码描述的能力。该框架涵盖Qwen、DeepSeek、Phi、LLaMA和Mistral等多种规模、架构和训练方式的Transformer模型。NLD-LLM的核心是综合的提示设计策略，包括标准化格式、明确的任务指引和专门的NLD提示，确保评估的公平性与一致性。此外，采用迭代优化流程提升输出质量并评估模型适应能力。通过语义与结构指标分析表明，提示工程显著影响模型效果，当有良好提示支持时，小型模型常表现出与大型模型相当的竞争力。
原文摘要 · Abstract (English)
Natural Language Description (NLD) is a Natural Language Processing (NLP) task that requires models to generate structured and meaningful outputs from natural language inputs. In this work, we propose NLD-LLM, a systematic NLP framework to evaluate the performance of language models to generate accurate and concise source code descriptions. This framework incorporates a diverse set of transformer models, including Qwen, DeepSeek, Phi, LLaMA, and Mistral, spanning various sizes, architectures, and training approaches. Central to NLD-LLM is a comprehensive prompt design strategy that includes standardized formatting, clear task guidance, and NLD prompting, ensuring fair and consistent evaluation. Additionally, we apply an iterative refinement process to improve output's quality and assess the model's adaptability. Using semantic and structural metrics, our analysis demonstrates that prompt engineering significantly impacts the effectiveness of the model such that smaller models often performing competitively when supported by well-crafted prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。