用大模型自动分类GitHub项目说明文件内容，提升代码库可发现性。
LLM-based Content Classification Approach for GitHub Repositories by the README Files
- 基于BERT、DistilBERT等模型微调，识别README文件各部分内容。
- 在4226个样本上达到0.98的综合F1分数，优于现有方法。
- 采用轻量微调技术LoRA，节省资源且性能损失小，适合实际部署。
GitHub是全球最流行的代码存储与协作平台，每个仓库都配有README文件。根据GitHub建议，README应包含项目相关信息以促进使用和改进，但许多维护者忽视此规范，限制了仓库的潜力。本研究提出，README内容的完整性显著影响其被采纳与利用的程度。大型语言模型（LLMs）在文本分类、生成等任务中表现优异。本文构建了一个方法，对三种编码器型预训练模型（BERT、DistilBERT、RoBERTa）进行微调，实现对README文件段落内容的自动分类。该方法基于包含4226个段落的黄金标准数据集，整体F1分数达0.98，超越现有最优方法。同时探索了参数高效微调技术（如LoRA），证明其在不明显牺牲性能的前提下，为全量微调提供了经济替代方案。结果表明，利用LLM可有效构建自动化分类器，助力提升GitHub仓库的识别度与使用潜力。
原文摘要 · Abstract (English)
GitHub is the world's most popular platform for storing, sharing, and managing code. Every GitHub repository has a README file associated with it. The README files should contain project-related information as per the recommendations of GitHub to support the usage and improvement of repositories. However, GitHub repository owners sometimes neglected these recommendations. This prevents a GitHub repository from reaching its full potential. This research posits that the comprehensiveness of a GitHub repository's README file significantly influences its adoption and utilization, with a lack of detail potentially hindering its full potential for widespread engagement and impact within the research community. Large Language Models (LLMs) have shown great performance in many text-based tasks including text classification, text generation, text summarization and text translation. In this study, an approach is developed to fine-tune LLMs for automatically classifying different sections of GitHub README files. Three encoder-only LLMs are utilized, including BERT, DistilBERT and RoBERTa. These pre-trained models are then fine-tuned based on a gold-standard dataset consisting of 4226 README file sections. This approach outperforms current state-of-the-art methods and has achieved an overall F1 score of 0.98. Moreover, we have also investigated the use of Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) and shown an economical alternative to full fine-tuning without compromising much performance. The results demonstrate the potential of using LLMs in designing an automatic classifier for categorizing the content of GitHub README files. Consequently, this study contributes to the development of automated tools for GitHub repositories to improve their identifications and potential usages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。