research area Methods and Algorithms, research area Speech and Natural Language Processingconference COLM
content type paperpublished July 2025

Overcoming Vocabulary Constraints with Pixel-level Fallback

AuthorsJonas F. Lotz†**, Hendra Setiawan, Stephan Peitz, Yova Kementchedjhieva‡

View publication

Subword tokenization requires balancing computational efficiency and vocabulary coverage, which often leads to suboptimal performance on languages and scripts not prioritized during training. We propose to augment pretrained language models with a vocabulary-free encoder that generates input embeddings from text rendered as pixels. Through experiments on English-centric language models, we demonstrate that our approach substantially improves machine translation performance and facilitates effective cross-lingual transfer, outperforming tokenizer-based methods. Furthermore, we find that pixel-based representations outperform byte-level approaches and standard vocabulary expansion. Our approach enhances the multilingual capabilities of monolingual language models without extensive retraining and reduces decoding latency via input compression.

  • † University of Copenhagen
  • ‡ Mohamed bin Zayed University of Artificial Intelligence
  • ** Work done while at Apple
Diagram of Hindi-to-English translation pipeline: left shows source encoding and generation process, right shows fallback network segmentation, patch rendering, and word embedding output.
Illustration of fallback network: text segmented, rendered into bigram patches, and embedded for input into decoder-only LLM.

Related readings and updates.

Cut Your Losses in Large-Vocabulary Language Models

February 7, 2025research area Methods and Algorithmsconference ICLR

As language models grow ever larger, so do their vocabularies. This has shifted the memory footprint of LLMs during training disproportionately to one single layer: the cross-entropy in the loss computation. Cross-entropy builds up a logit matrix with entries for each pair of input tokens and vocabulary items and, for small models, consumes an order of magnitude more memory than the rest of the LLM combined. We propose Cut Cross-Entropy (CCE), a...

Read more

Training a Tokenizer for Free with Private Federated Learning

May 23, 2022research area Privacy, research area Speech and Natural Language Processingconference ACL

Federated learning with differential privacy, i.e. private federated learning (PFL), makes it possible to train models on private data distributed across users’ devices without harming privacy. PFL is efficient for models, such as neural networks, that have a fixed number of parameters, and thus a fixed-dimensional gradient vector. Such models include neural-net language models, but not tokenizers, the topic of this work. Training a tokenizer...

Read more