The paper highlights a lopsided distribution of parameters in small language models: the embedding table, which maps tokens to vectors, can account for the majority of the model's size. In the case of Gemma 3 270M, the embedding table holds 168M of the 268M parameters. Because quantisation methods usually keep embeddings in 16-bit, the table also makes up most of the stored file, even after other weights are compressed.
The authors argue that this is wasteful, since the vocabulary is often overprovisioned relative to the model's capacity. They propose replacing the learned embedding table with a frozen, compact vocabulary of only 16 MB, which would drastically reduce the model's footprint. The idea is that a small model does not need a large, trainable embedding space; a fixed representation can suffice.
The paper does not claim this approach is universally better, but it suggests that for small and ternary models, the embedding is an overlooked bottleneck. If the method holds up, it could make such models far more practical for on-device or memory-constrained deployment, where file size and RAM usage are critical. Further details on the training procedure and downstream task performance are not covered in the abstract.