Search Results for author:
Found 12 papers, 1 papers with code
HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
In this paper, we showcase HLAT: a 7 billion parameter decoder-only LLM pre-trained using trn1 instances over 1. 8 trillion tokens.
EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence
We follow the global contrastive learning loss as introduced in SogCLR, and propose EMC$^2$ which utilizes an adaptive Metropolis-Hastings subroutine to generate hardness-aware negative samples in an online fashion during the optimization.
Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
Evaluated across a range of both masked and autoregressive LMs on benchmark GLUE tasks, MeZO-SVRG outperforms MeZO with up to 20% increase in test accuracies in both full- and partial-parameter fine-tuning settings.
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent
Since Adam was introduced, several novel adaptive optimizers for deep learning have been proposed.
Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate
In this paper, we introduce a novel subspace cubic regularized Newton method that achieves a dimension-independent global convergence rate of ${O}\left(\frac{1}{mk}+\frac{1}{k^2}\right)$ for solving convex optimization problems.
Contractive error feedback for gradient compression
On-device memory concerns in distributed deep learning have become severe due to (i) the growth of model size in multi-GPU training, and (ii) the wide adoption of deep neural networks for federated learning on IoT devices which have limited storage.
DS-FACTO: Doubly Separable Factorization Machines
Traditional algorithms for FM which work on a single-machine are not equipped to handle this scale and therefore, using a distributed algorithm to parallelize the computation across a cluster is inevitable.
Optimization on the Surface of the (Hyper)-Sphere
Thomson problem is a classical problem in physics to study how $n$ number of charged particles distribute themselves on the surface of a sphere of $k$ dimensions.
Extreme Stochastic Variational Inference: Distributed and Asynchronous
Moreover, it requires the parameters to fit in the memory of a single processor; this is problematic when the number of parameters is in billions.
DS-MLR: Exploiting Double Separability for Scaling up Distributed Multinomial Logistic Regression
Scaling multinomial logistic regression to datasets with very large number of data points and classes is challenging.
Ranking via Robust Binary Classification
We propose RoBiRank, a ranking algorithm that is motivated by observing a close connection between evaluation metrics for learning to rank and loss functions for robust classification.
Ranking via Robust Binary Classification and Parallel Parameter Estimation in Large-Scale Data
We propose RoBiRank, a ranking algorithm that is motivated by observing a close connection between evaluation metrics for learning to rank and loss functions for robust classification.
