用GPT-4o生成网页存档元数据，成本降99.9%，但质量仍不如人工。
Web Archives Metadata Generation with GPT-4o: Challenges and Insights
- 用提示工程生成标题摘要，结合数据压缩技术降低处理开销。
- 元数据生成成本减少99.9%，效率显著提升，但存在幻觉与翻译问题。
- 适合面临人力成本高的档案机构，可作人工辅助而非替代。
当前网页存档的元数据创建依赖人工，耗时且昂贵。本文探索在新加坡网页存档中使用GPT-4o进行元数据生成，关注可扩展性、效率与成本效益。通过数据缩减技术处理112个WARC文件，实现元数据生成成本降低99.9%。采用提示工程生成标题与摘要，分别通过编辑距离（Levenshtein Distance）和BERTScore进行内在评估，以及由人工编目员使用McNemar检验进行外在评估。结果显示，尽管该方法在成本与效率上优势明显，但人工标注仍具质量优势。研究识别出内容错误、幻觉和翻译问题等关键挑战，表明大语言模型应作为人工编目的补充而非替代。未来工作将聚焦于优化提示、改进内容过滤，并通过小型模型实验解决隐私问题。代码已开源，供面临类似挑战的机构使用。
原文摘要 · Abstract (English)
Current metadata creation for web archives is time consuming and costly due to reliance on human effort. This paper explores the use of gpt-4o for metadata generation within the Web Archive Singapore, focusing on scalability, efficiency, and cost effectiveness. We processed 112 Web ARChive (WARC) files using data reduction techniques, achieving a notable 99.9% reduction in metadata generation costs. By prompt engineering, we generated titles and abstracts, which were evaluated both intrinsically using Levenshtein Distance and BERTScore, and extrinsically with human cataloguers using McNemar's test. Results indicate that while our method offers significant cost savings and efficiency gains, human curated metadata maintains an edge in quality. The study identifies key challenges including content inaccuracies, hallucinations, and translation issues, suggesting that Large Language Models (LLMs) should serve as complements rather than replacements for human cataloguers. Future work will focus on refining prompts, improving content filtering, and addressing privacy concerns through experimentation with smaller models. This research advances the integration of LLMs in web archiving, offering valuable insights into their current capabilities and outlining directions for future enhancements. The code is available at https://github.com/masamune-prog/warc2summary for further development and use by institutions facing similar challenges.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。