用共享语义单元压缩词元表示，参数量仅0.5%仍保持95%性能
Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics
- 将词元拆解为共享语义组件，通过积量化实现压缩表示
- 参数压缩至0.4%-0.5%仍保持超95%任务性能
- 适用于多语言、跨领域及生成任务，适合资源受限场景
标准语言模型为每个词元使用独立的完整嵌入，可能限制其捕捉词语多重语义的能力。本文提出聚合语义分组（ASG），利用积量化（PQ）构建可组合的语义表示。在mBERT、XLM-R、mT5等Transformer架构上应用，并在自然语言推断、命名实体识别、问答等任务，以及生物医学领域的BC5CDR基准（使用BioBERT）上评估。结果表明，通过ASG进行组合式表示可在保持95%以上基线性能的前提下，实现0.4%–0.5%的嵌入参数极端压缩，即使在生成任务中也有效，且支持跨语言迁移与领域适配。研究验证了词元可被建模为共享语义单元的组合，提供了简单而有效的紧凑高语义模型实现路径。
原文摘要 · Abstract (English)
Standard language models employ unique, monolithic embeddings for each token, potentially limiting their ability to capture the multifaceted nature of word meanings. We investigate whether tokens can be more effectively represented through a compositional structure that accumulates diverse semantic facets. To explore this, we propose Aggregate Semantic Grouping (ASG), a novel approach leveraging Product Quantization (PQ). We apply ASG to standard transformer architectures (mBERT, XLM-R, mT5) and evaluate this representational scheme across diverse tasks (NLI, NER, QA), as well as a biomedical domain-specific benchmark (BC5CDR) using BioBERT. Our findings demonstrate that representing tokens compositionally via ASG achieves extreme compression in embedding parameters (0.4--0.5\%) while maintaining $>$95\% task performance relative to the base model, even in generative tasks and extends to both cross lingual transfer and domain-specific settings. These results validate the principle that tokens can be effectively modeled as combinations of shared semantic building blocks. ASG offers a simple yet concrete method for achieving this, showcasing how compositional representations can capture linguistic richness while enabling compact yet semantically rich models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。