CHERRY用选择性监督压缩模型，实现高效韩国语大模型。
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
- 只监督决定答案的词元，共享权重处理其余部分
- 48层模型压缩至6个块，参数量227M损失仅2.934
- 适合追求高效、主权化语言模型的研究与应用
前沿语言能力通常依赖大量算力；而CHERRY提出不同路径。其基于一个原则：仅监督决定答案的词元，让共享权重承载其余部分。在相同算力下，显现出清晰可复现的差异——选择性监督保留未见区分能力，但崩溃自由生成；全序列锚点仅恢复部分差距。同一信号驱动融合后合并的循环表征增益机制，将48层压缩至6个独特模块，接近密集模型性能（227M参数，损失2.934；对比566M密集模型损失2.926），并通过MoEE融合达到2.789损失。该循环压缩方向与同期前沿环形MoE工作独立契合，预计可扩展至前沿规模（尚未测量）。通过双词元监督引入元认知：在200个保留韩国指令/类型上，κ>0.82，自修正率从12%提升至47%，越狱成功率从23%降至4%，且在12亿参数模型上保持97.6%损失保留。10亿→137亿参数消融实验表明操作符绑定瓶颈源于容量（10亿查找对比137亿H-PRESERVE）。18亿参数模型在CyberMetric任务上达到人类专家水平（75.0%优于30专家平均72.24%）。发布的1.8亿参数分词器比Gemma-4更高效9.2%；从头训练的120亿参数模型配备主权韩国分词器（词表131,037）。在政府运营的K-AI韩国大模型榜单中，1.8亿版本领先HLE(Ko)列（0.123对0.077；总分51/78）。前沿层面，可在单个120GB加速器上微调并服务适配后的1220亿参数模型（峰值占用83GB）。各成员出处明确，架构与配方均为原创，所有主张均对应已发布验证数据。
原文摘要 · Abstract (English)
Frontier language capability is usually bought with frontier compute; CHERRY shows a different trade. It is a sovereign Korean model family built on one principle: supervise the tokens that decide the answer, and let shared weights carry the rest. Under matched compute this exposes a sharp, reproducible dissociation---selected-token supervision preserves held-out discrimination yet collapses free generation, and a full-sequence anchor recovers only part of the gap. The same signal drives a heal-after-merge recurrent-representational-yield loop that collapses 48 layers to 6 unique blocks at near-dense parity (227M at loss 2.934 vs a 566M dense model at 2.926) and composes them by MoEE fusion (2.789)---a recurrent-compression direction independently pursued by concurrent frontier looped-MoE work, which we project (not yet measure) to frontier scale. It also installs metacognition from two-token supervision (200 held-out KO prompts/type, kappa>0.82, +/-6.9pp): self-correction 12->47% and jailbreak 23->4% at 97.6% loss-retention on 1.2B, with a pre-registered 1B->13.7B ablation localising the operand-binding limit to capacity (1B lookup vs 13.7B H-PRESERVE); and it specializes a 1.8B model to reported human-expert level on CyberMetric (75.0% vs a 30-expert average 72.24%). The released 1.8B tokenizer is 9.2% more Korean-efficient than Gemma-4; the from-scratch 12B adds a sovereign Korean tokenizer (vocab 131,037). On the government-operated K-AI Korean-LLM leaderboard the released 1.8B leads the HLE(Ko) column (0.123 vs 0.077; overall 51/78). At the frontier we fine-tune and serve an adapted 122B model on one 120GB accelerator (83GB measured peak). Provenance differs by member and we state it exactly; the architecture and recipe are ours throughout, and every claim is bound to a released curated measurement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。