LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT

13 Jan 2021 • Mohammed Azhan • Mohammad Ahmad

In our paper, we present Deep Learning models with a layer differentiated training method which were used for the SHARED [email protected] CONSTRAINT 2021 sub-tasks COVID19 Fake News Detection in English and Hostile Post Detection in Hindi. We propose a Layer Differentiated training procedure for training a pre-trained ULMFiT arXiv:1801.06146 model... (read more)

PDF Abstract

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

METHOD	TYPE
Sigmoid Activation	Activation Functions
Tanh Activation	Activation Functions
Dropout	Regularization
Embedding Dropout	Regularization
Layer Normalization	Normalization
Softmax	Output Functions
Attention Dropout	Regularization
LSTM	Recurrent Neural Networks
Weight Tying	Parameter Sharing
DropConnect	Regularization
Residual Connection	Skip Connections
GELU	Activation Functions
Adam	Stochastic Optimization
Weight Decay	Regularization
Linear Warmup With Linear Decay	Learning Rate Schedules
Activation Regularization	Regularization
Dense Connections	Feedforward Networks
Temporal Activation Regularization	Regularization
Scaled Dot-Product Attention	Attention Mechanisms
Slanted Triangular Learning Rates	Learning Rate Schedules
WordPiece	Subword Segmentation
Multi-Head Attention	Attention Modules
BERT	Language Models
Variational Dropout	Regularization
AWD-LSTM	Recurrent Neural Networks
Discriminative Fine-Tuning	Fine-Tuning
ULMFiT	Language Models
RoBERTa	Transformers

METHOD

TYPE

Sigmoid Activation