联合训练可让小模型学得更好，关键在任务搭配是否合适。
Small Models, Smarter Learning: The Power of Joint Task Training
- 通过控制实验发现，特定任务组合能降低模型参数需求2-7倍。
- 成功联合训练使模型学会数字排序、奇偶性等结构化表示。
- 适合想用小模型高效学习的算法设计者和课程规划者。
多任务学习提升泛化能力，但何时能减少所需模型容量？我们以嵌套算术（ListOps）和置换群为受控测试平台，系统研究联合训练对学习过渡的影响，即任务可被学习的最小模型规模。某些任务组合显著降低模型规模要求：将简单任务（如MAX、MIN、PROD）与困难任务（如模加法、置换乘积）结合，可实现2-7倍的参数减少。关键发现是：当任务结构相似（如加法与交替符号非加法）或缺乏共享计算原语时，协同效应失效。主成分分析显示，成功联合训练诱导出单任务模型中不存在的结构化数表示（如排序、奇偶性、模结构）。迁移实验验证这些表示具有因果性：在简单任务上预训练的模型，可在7倍更小的规模下学会加法。结果表明，任务兼容性而非单纯多样性决定联合训练能否降低容量需求，为课程设计提供量化指导。
原文摘要 · Abstract (English)
Multi-task learning improves generalization, but when does it reduce the model capacity required to learn? We provide a systematic study of how joint training affects the learning transition, the minimum model size at which a task can be learned, using nested arithmetic (ListOps) and permutation groups as controlled testbeds. Certain task pairings dramatically reduce model size requirements: combining easy operations (MAX, MIN, PROD) with hard ones (modular addition, permutation products) enables learning with 2-7 times fewer parameters. Crucially, we also identify when synergies fail: pairing structurally similar hard tasks (e.g., ADD with alternating-sign NADD) provides no benefit, nor does pairing tasks lacking shared computational primitives. PCA of learned embeddings reveals that successful joint training induces structured number representations (ordering, parity, modular structure) absent in single-task models. Transfer experiments confirm these representations are causal: models pretrained on easy tasks learn addition at 7 times smaller sizes. Our results establish that task compatibility, not mere diversity, determines whether joint training reduces capacity requirements, providing quantitative guidance for curriculum design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。