Search Results for author:
Found 8 papers, 3 papers with code
SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
The advancement of deep learning has led to the emergence of Mixture-of-Experts (MoEs) models, known for their dynamic allocation of computational resources based on input.
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost.
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
Scalable and Efficient MoE Training for Multitask Multilingual Models
By combining the efficient system and training methods, we are able to significantly scale up large multitask multilingual models for language generation which results in a great improvement in model accuracy.
Improving Multilingual Translation by Representation and Gradient Regularization
At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients.
Discovering Representation Sprachbund For Multilingual Pre-Training
Then we cluster all the target languages into multiple groups and name each group as a representation sprachbund.
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
