Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

PDF Abstract ICLR 2022 PDF ICLR 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Linguistic Acceptability CoLA Charformer-Tall Accuracy 51.8% # 22
Semantic Textual Similarity MRPC Charformer-Tall Accuracy 87.5% # 16
F1 91.4 # 6
Natural Language Inference MultiNLI Charformer-Tall Matched 83.7 # 23
Mismatched 84.4 # 17
Natural Language Inference QNLI Charformer-Tall Accuracy 91.0% # 22
Paraphrase Identification Quora Question Pairs Charformer-Tall Accuracy 91.4 # 2
F1 88.5 # 2
Sentiment Analysis SST-2 Binary classification Charformer-Base Accuracy 91.6 # 38
Semantic Textual Similarity STS Benchmark Charformer-Tall Pearson Correlation 0.873 # 20