Search Results for author:
Found 3 papers, 2 papers with code
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without significant additional computational cost.
PAD-Net: An Efficient Framework for Dynamic Networks
The main contributions of our work are challenging the basic commonsense in dynamic networks and proposing a partially dynamic network, namely PAD-Net, to transform the redundant dynamic parameters into static ones.
Vega-MT: The JD Explore Academy Translation System for WMT22
As for model sizes, we scale the Transformer-Big up to the extremely large model that owns nearly 4. 7 Billion parameters, to fully enhance the model capacity for our Vega-MT.
Ranked #1 on
Machine Translation
on WMT 2022 English-Russian
