ByT5: Towards a token-free future with pre-trained byte-to-byte models

Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units. Encoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from the model... Token-free models that instead operate directly on raw text (bytes or characters) have many benefits: they can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We carefully characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments. read more

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Extreme Summarization GEM-XSum mT5 BLEU score 14.3 # 2
Extreme Summarization GEM-XSum ByT5 BLEU score 15.3 # 1
Cross-Lingual Question Answering MLQA ByT5 XXL EM 54.9 # 1
F1 71.6 # 1
Cross-Lingual Paraphrase Identification PAWS-X ByT5 Small Accuracy 84 # 4
Cross-Lingual Paraphrase Identification PAWS-X ByT5 XXL Accuracy 90.1 # 1
Question Answering TweetQA ByT5 ROUGE-L 75.7 # 1
Question Answering TweetQA ByT5 (small) BLEU-1 72.0 # 1
Question Answering TweetQA mT5 BLEU-1 70.8 # 2
ROUGE-L 74.3 # 2
Cross-Lingual Question Answering TyDiQA-GoldP ByT5 XXL EM 60.0 # 1
F1 75.3 # 1
Cross-Lingual NER WikiAnn NER ByT5 XXL F1 67.7 # 1
Cross-Lingual Natural Language Inference XNLI ByT5 XXL Accuracy 83.7 # 1
Cross-Lingual Natural Language Inference XNLI ByT5 Small Accuracy 69.1 # 4
Cross-Lingual Question Answering XQuAD ByT5 XXL EM 63.6 # 1
F1 79.7 # 1

Methods