Search Results for author:
Found 18 papers, 6 papers with code
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance.
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation
Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions.
Scaling Laws for Multilingual Neural Machine Translation
Through a novel joint scaling law formulation, we compute the effective number of parameters allocated to each language pair and examine the role of language similarity in the scaling behavior of our models.
Binarized Neural Machine Translation
In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind.
Do Current Multi-Task Optimization Methods in Deep Learning Even Help?
Recent research has proposed a series of specialized optimization algorithms for deep multi-task models.
Adaptive Gradient Methods at the Edge of Stability
Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning.
Data Scaling Laws in NMT: The Effect of Noise and Architecture
In this work, we study the effect of varying the architecture and training data quality on the data scaling properties of Neural Machine Translation (NMT).
Examining Scaling and Transfer of Language Model Architectures for Machine Translation
Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing.
A Loss Curvature Perspective on Training Instability in Deep Learning
In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics.
A Loss Curvature Perspective on Training Instabilities of Deep Learning Models
In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics.
Scaling Laws for Neural Machine Translation
We show that cross-entropy loss as a function of model size follows a certain scaling law.
When Do Neural Networks Outperform Kernel Methods?
Recent empirical work showed that, for some classification tasks, RKHS methods can replace NNs without a large loss in performance.
Limitations of Lazy Training of Two-layers Neural Network
We study the supervised learning problem under either of the following two models: (1) Feature vectors x_i are d-dimensional Gaussian and responses are y_i = f_*(x_i) for f_* an unknown quadratic function; (2) Feature vectors x_i are distributed as a mixture of two d-dimensional centered Gaussians, and y_i's are the corresponding class labels.
Limitations of Lazy Training of Two-layers Neural Networks
We study the supervised learning problem under either of the following two models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic function; (2) Feature vectors ${\boldsymbol x}_i$ are distributed as a mixture of two $d$-dimensional centered Gaussians, and $y_i$'s are the corresponding class labels.
The Effect of Network Depth on the Optimization Landscape
It is well-known that deeper neural networks are harder to train than shallower ones.
Linearized two-layers neural networks in high dimension
Both these approaches can also be regarded as randomized approximations of kernel ridge regression (with respect to different kernels), and enjoy universal approximation properties when the number of neurons $N$ diverges, for a fixed dimension $d$.
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process.
An Instability in Variational Inference for Topic Models
Namely, for certain regimes of the model parameters, variational inference outputs a non-trivial decomposition into topics.
