Pico让小模型研究像科学实验一样可验证，支持快速迭代设计。
Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
- 提供模块化工具链，可精准修改模型架构或训练方式
- 通过标准化基线模型实现可复现的实验对比
- 适合想系统优化小模型的研究者和工程师
构建语言模型，尤其是小型和中等规模的模型，仍更像艺术而非科学。尽管大模型常因规模提升而表现更好，但许多设计选择为何有效仍不明确。对于小模型而言，这种不确定性更为严重：参数预算紧张，每个决策都至关重要，但研究人员仍缺乏系统、科学的方法来测试和优化新想法。我们提出 Pico——一个轻量级、模块化的框架，支持针对小中型语言模型的系统性、假设驱动式研究。Pico 包含两个库，共同构成一个实用的实验沙盒，使研究者能够对模型架构或训练流程进行精准改动，并直接观察其对模型行为的影响。为支持可复现实验，我们还发布了在标准条件下训练的基准模型 pico-decoder，已开源供社区使用。案例研究展示了 Pico 如何支持小模型的迭代设计与分析。
原文摘要 · Abstract (English)
Building language models (LMs), especially small and medium ones, remains more art than science. While large LMs often improve by sheer scale, it is still unclear why many design choices work. For small LMs, this uncertainty is more limiting: tight parameter budgets make each decision critical, yet researchers still lack systematic, scientific ways to test and refine new ideas. We introduce Pico, a lightweight, modular framework that enables systematic, hypothesis-driven research for small and medium-scale language model development. Pico consists of two libraries that together provide a practical sandbox where researchers can make targeted changes to a model's architecture or training procedures and directly observe their effects on the model's behavior. To support reproducible experimentation, we also release a suite of baseline models, pico-decoder, trained under standardized conditions and open-sourced for the community. Case studies highlight how Pico can support iterative small LM design and analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。