Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing

We introduce Trankit, a light-weight Transformer-based Toolkit for multilingual Natural Language Processing (NLP). It provides a trainable pipeline for fundamental NLP tasks over 100 languages, and 90 pretrained pipelines for 56 languages. Built on a state-of-the-art pretrained language model, Trankit significantly outperforms prior multilingual NLP pipelines over sentence segmentation, part-of-speech tagging, morphological feature tagging, and dependency parsing while maintaining competitive performance for tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. Despite the use of a large pretrained transformer, our toolkit is still efficient in memory usage and speed. This is achieved by our novel plug-and-play mechanism with Adapters where a multilingual pretrained transformer is shared across pipelines for different languages. Our toolkit along with pretrained models and code are publicly available at: A demo website for our toolkit is also available at: Finally, we create a demo video for Trankit at:

PDF Abstract EACL 2021 PDF EACL 2021 Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Sentence segmentation UD2.5 test Trankit Macro-averaged F1 91.82 # 1
Dependency Parsing UD2.5 test Stanza Macro-averaged F1 83.06 # 2
Part-Of-Speech Tagging UD2.5 test Stanza Macro-averaged F1 94.21 # 2
Sentence segmentation UD2.5 test Stanza Macro-averaged F1 88.58 # 2
Dependency Parsing UD2.5 test Trankit Macro-averaged F1 87.06 # 1
Part-Of-Speech Tagging UD2.5 test Trankit Macro-averaged F1 95.65 # 1