150万参数模型直接分析原始字节，一次预测七类内容属性。
pico-type: A 1.5M-Parameter Byte-Level Multi-Head Content Classifier
- 无需分词器，直接在字节层面用卷积与注意力网络提取特征。
- 代码语言识别准确率60.3%，文本语言达98.2%，显著优于纯合成数据训练。
- 轻量级设计支持快速部署，适合本地安全扫描或隐私检测场景。
我们提出pico-type，一个约150万参数的字节级多头内容分类器，能在单次前向传播中同时预测七种内容属性：粗粒度类型（12类）、模态（8类）、子类型（24类）、代码语言（62类）、文本语言（30类）、文件MIME类型（90类）及风险标签（6类多标签：API密钥、JWT、密码、邮箱、电话、SSH密钥）。模型直接处理原始UTF-8字节，不依赖分词器、子词词汇表或预训练嵌入。架构包含学习型字节嵌入、三个感受野递增的卷积块、两个带旋转位置编码的双向注意力层，以及统计池化层，连接七个套娃式分类头。提供四种层级变体（tiny/small/base/pro），共享主干网络并截取16至576维表示，支持小于210KB的ONNX导出和10毫秒内CPU推理。在合成模板与真实数据混合训练下（含8709个GitHub代码样本、5000篇维基百科文章），在The Heap基准上代码语言识别准确率达60.3%（24语言），相比纯合成基线提升57个百分点；在维基百科上文本语言准确率达98.2%，提升79个百分点。格式相关头部（粗粒度、模态、子类型、file_mime、risk）在合成基准上保持100%准确。模型、代码及预训练权重已开源，采用Apache 2.0许可。
原文摘要 · Abstract (English)
We introduce pico-type, a byte-level multi-head content classifier with approximately 1.5 million parameters that simultaneously predicts seven content properties from raw UTF-8 bytes in a single forward pass. Operating directly at the byte level -- no tokenizer, no subword vocabulary, no pretrained embeddings -- pico-type classifies coarse type (12 classes), modality (8), subtype (24), code language (62), text language (30), file MIME type (90), and risk flags (6-label multi-label: API keys, JWTs, passwords, emails, phone numbers, SSH keys). The architecture combines a learned byte embedding, three convolutional blocks with growing receptive fields, two bidirectional attention layers with rotary position encodings, and a statistical pooling layer feeding seven Matryoshka-style classification heads. Four tiered variants (tiny/small/base/pro) share the same trunk with sliced representations from 16 to 576 dimensions, yielding ONNX exports under 210 KB and CPU inference under 10 ms. Trained on a mixture of synthetic templates and real-world data (8709 GitHub code samples, 5000 Wikipedia articles), pico-type achieves 60.3 percent code language accuracy on The Heap benchmark (24 languages) and 98.2 percent text language accuracy on Wikipedia (30 languages) -- improvements of +57 and +79 percentage points respectively over the synthetic-only baseline. Format-based heads (coarse, modality, subtype, file_mime, risk) maintain 100 percent accuracy on synthetic benchmarks. The model, code, and pretrained weights are released under Apache 2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。