Search Results for author:
Found 23 papers, 9 papers with code
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
Adaptive Gradient Methods at the Edge of Stability
Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning.
Pre-training helps Bayesian optimization too
Contrary to a common belief that BO is suited to optimizing black-box functions, it actually requires domain knowledge on characteristics of those functions to deploy BO successfully.
AI system for fetal ultrasound in low-resource settings
Using a simplified sweep protocol with real-time AI feedback on sweep quality, we have demonstrated the generalization of model performance to minimally trained novice ultrasound operators using low cost ultrasound devices with on-device AI integration.
Predicting the utility of search spaces for black-box optimization:a simple, budget-aware approach
For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable.
Pre-trained Gaussian processes for Bayesian optimization
Contrary to a common expectation that BO is suited to optimizing black-box functions, it actually requires domain knowledge about those functions to deploy BO successfully.
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements.
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes.
Ranked #1 on
Question Answering
on SQuAD1.1
(Hardware Burden metric)
On Empirical Comparisons of Optimizers for Deep Learning
In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent.
Faster Neural Network Training with Data Echoing
In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training.
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns.
Measuring the Effects of Data Parallelism on Neural Network Training
Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes.
The Importance of Generation Order in Language Modeling
Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks.
Motivating the Rules of the Game for Adversarial Example Research
Advances in machine learning have led to broad deployment of systems with impressive performance on important problems.
Embedding Text in Hyperbolic Spaces
Ideally, we could incorporate our prior knowledge of this hierarchical structure into unsupervised learning algorithms that work on text data.
Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
We present a simple and powerful algorithm for parallel black box optimization called Successive Halving and Classification (SHAC).
Large scale distributed neural network training through online distillation
Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made.
Neural Message Passing for Quantum Chemistry
Supervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science.
Ranked #5 on
Graph Regression
on ZINC 100k
Detecting Cancer Metastases on Gigapixel Pathology Images
At 8 false positives per image, we detect 92. 4% of the tumors, relative to 82. 7% by the previous best automated approach.
Ranked #2 on
Medical Object Detection
on Barrett’s Esophagus
Machine learning prediction errors better than DFT accuracy
We investigate the impact of choosing regressors and molecular representations for the construction of fast machine learning (ML) models of thirteen electronic ground-state properties of organic molecules.
Ranked #18 on
Formation Energy
on QM9
Incorporating Side Information in Probabilistic Matrix Factorization with Gaussian Processes
Probabilistic matrix factorization (PMF) is a powerful method for modeling data associ- ated with pairwise relationships, Finding use in collaborative Filtering, computational bi- ology, and document analysis, among other areas.
Multi-task Neural Networks for QSAR Predictions
Although artificial neural networks have occasionally been used for Quantitative Structure-Activity/Property Relationship (QSAR/QSPR) studies in the past, the literature has of late been dominated by other machine learning techniques such as random forests.
Improvements to deep convolutional neural networks for LVCSR
We find that with these improvements, particularly with fMLLR and dropout, we are able to achieve an additional 2-3% relative improvement in WER on a 50-hour Broadcast News task over our previous best CNN baseline.
