UL2 is a unified framework for pretraining models that are universally effective across datasets and setups. UL2 uses Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes.
Source: UL2: Unifying Language Learning ParadigmsPaper | Code | Results | Date | Stars |
---|
Task | Papers | Share |
---|---|---|
Question Answering | 5 | 16.13% |
Retrieval | 3 | 9.68% |
Language Modelling | 2 | 6.45% |
Relation Extraction | 2 | 6.45% |
Natural Language Inference | 2 | 6.45% |
Decoder | 1 | 3.23% |
Natural Language Understanding | 1 | 3.23% |
Named Entity Recognition (NER) | 1 | 3.23% |
NER | 1 | 3.23% |
Component | Type |
|
---|---|---|
🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |