arXiv:2607.21284cs.CL2026-07

小模型高效提取新闻内容,兼顾准确率与部署成本。

news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling

论文配图:news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling
图 1 · 摘自论文原文
  • 用微调的小型长文本模型处理网页结构化提取
  • 在多类任务中比基线提升4.8~6.1点指标
  • 适合需要低成本高精度新闻爬取的场景

从新闻页面提取结构化内容仍面临布局不一、标记混乱和大量无关信息(如导航栏、广告)等挑战。基于规则的爬虫虽精度高,但需手动配置难以泛化;大模型灵活但计算开销大。本文提出news-crawler-LM,一个在Fundus新闻爬取库中高质量人工验证数据上微调的小型长上下文语言模型。该模型将原始HTML转换为纯文本和结构化JSON,包含标题、作者、发布日期、正文等字段。实验显示,news-crawler-LM在HTML转Markdown和HTML转JSON任务中分别提升4.8 BLEU和6.1 METEOR,以及2.2 BLEU和4.1 METEOR;但在未见出版社的HTML转纯文本任务上仅略优于其他规则解析库。所有模型与资源已开源。

原文摘要 · Abstract (English)

Extracting structured content from news pages remains challenging due to heterogeneous HTML layouts, inconsistent markup, and substantial boilerplate such as navigation elements and advertisements. Rule-based news crawlers can achieve high extraction accuracy by encoding site-specific structure, but require manual configuration in order to generalize to new publishers. Large language models provide a more flexible alternative by reducing the need for handcrafted rules, but their high computational cost limits practical deployment. In this paper, we introduce news-crawler-LM, a small long-context language model fine-tuned on high-quality, human-validated extractions from the Fundus news-crawling library. Our model converts raw HTML into plaintext and structured JSON, including fields such as headline, author, publication date, and article body. In our experiments, news-crawler-LM outperforms strong baselines in HTML-to-Markdown and HTML-to-JSON extraction, improving performance by +4.8 BLEU and +6.1 METEOR in the HTML-to-Markdown task, and by +2.2 BLEU and +4.1 METEOR in the HTML-to-JSON task. However, we also observe that our model only slightly better compared to other rule-based parsing libraries on the HTML-to-plaintext task in evaluations on previously unseen publishers. We release all models and artifacts to the research community.

新闻抽取小模型长文本网页爬取

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。