Search Results for author:
Found 28 papers, 13 papers with code
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on TriviaQA
(using extra training data)
Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions
Adaptive regularization methods that exploit more than the diagonal entries exhibit state of the art performance for many tasks, but can be prohibitive in terms of memory and running time.
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation
In this work, we propose a novel approach for layerwise representation learning of a trained neural network.
On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models
For industrial-scale advertising systems, prediction of ad click-through rate (CTR) is a central problem.
N-Grammer: Augmenting Transformers with latent n-grams
Transformer models have recently emerged as one of the foundational models in natural language processing, and as a byproduct, there is significant recent interest and investment in scaling these models.
Ranked #5 on
Language Modelling
on C4
Learning from Randomly Initialized Neural Network Features
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation.
Step-size Adaptation Using Exponentiated Gradient Updates
In this paper, we update the step-size scale and the gain variables with exponentiated gradient updates instead.
Learning Rate Grafting: Transferability of Optimizer Tuning
In the empirical science of training large neural networks, the learning rate schedule is a notoriously challenging-to-tune hyperparameter, which can depend on all other properties (architecture, optimizer, batch size, dataset, regularization, ...) of the problem.
Efficiently Identifying Task Groupings for Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Large-Scale Differentially Private BERT
In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD).
LocoProp: Enhancing BackProp via Local Loss Optimization
Second-order methods have shown state-of-the-art performance for optimizing deep neural networks.
Knowledge distillation: A good teacher is patient and consistent
In particular, we uncover that there are certain implicit design choices, which may drastically affect the effectiveness of distillation.
Ranked #419 on
Image Classification
on ImageNet
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes.
Ranked #1 on
Question Answering
on SQuAD1.1
(Hardware Burden metric)
Information Transfer in Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Towards Practical Second Order Optimization for Deep Learning
Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent.
Measuring and Harnessing Transference in Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Stochastic Optimization with Laggard Data Pipelines
State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes.
Disentangling Adaptive Gradient Methods from Learning Rates
We investigate several confounding factors in the evaluation of optimization algorithms for deep learning.
Scalable Second Order Optimization for Deep Learning
Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent.
Revisiting the Generalization of Adaptive Gradient Methods
A commonplace belief in the machine learning community is that using adaptive gradient methods hurts generalization.
Memory Efficient Adaptive Optimization
Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling.
Robust Bi-Tempered Logistic Loss Based on Bregman Divergences
We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Memory-Efficient Adaptive Optimization
Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling.
Ranked #31 on
Machine Translation
on WMT2014 English-French
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank
We propose TensorFlow Ranking, the first open source library for solving large-scale ranking problems in a deep learning framework.
Large scale distributed neural network training through online distillation
Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made.
Wide & Deep Learning for Recommender Systems
Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort.
Ranked #2 on
Click-Through Rate Prediction
on Bing News
