构建了覆盖1900种语言的超大规模字符频数数据集，支持跨语言、跨时段分析。
FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text
- 从57TB网页文本中提取96万亿字符，按语言与年份统计字符频率
- 覆盖2013至2025年，支持细粒度时间演化分析
- 保留真实多语言特征，适合语言学、NLP与跨文化研究者使用
我们提出FineFreq，一个基于FineWeb和FineWeb2语料库构建的大规模多语言字符频率数据集，涵盖超过1900种语言，时间跨度为2013至2025年。该数据集包含从57TB压缩文本中处理的96万亿字符的频次统计，每种语言均提供字符级统计信息，包括聚合频率与逐年频率，支持细粒度的时间分析。数据集保留了自然存在的多语言特征，如跨文字借用、表情符号和缩略词，未施加人工过滤。每个字符条目包含Unicode元数据（类别、脚本、区块），便于下游特定领域或其它筛选分析。完整数据集以CSV和Parquet格式发布，附带元数据，可在GitHub和HuggingFace获取。
原文摘要 · Abstract (English)
We present FineFreq, a large-scale multilingual character frequency dataset derived from the FineWeb and FineWeb2 corpora, covering over 1900 languages and spanning 2013-2025. The dataset contains frequency counts for 96 trillion characters processed from 57 TB of compressed text. For each language, FineFreq provides per-character statistics with aggregate and year-level frequencies, allowing fine-grained temporal analysis. The dataset preserves naturally occurring multilingual features such as cross-script borrowings, emoji, and acronyms without applying artificial filtering. Each character entry includes Unicode metadata (category, script, block), enabling domain-specific or other downstream filtering and analysis. The full dataset is released in both CSV and Parquet formats, with associated metadata, available on GitHub and HuggingFace. https://github.com/Bin-2/FineFreq
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。