Search Results for author:
Found 42 papers, 15 papers with code
Evolving Machine Learning Algorithms From Scratch
However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces.
Brainformers: Trading Simplicity for Efficiency
Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions.
Rationale-Augmented Ensembles in Language Models
Recent research has shown that rationales, or step-by-step chains of thought, can be used to improve performance in multi-step reasoning tasks.
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks.
Ranked #28 on
Arithmetic Reasoning
on GSM8K
TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets
The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints including latency, compute, energy consumption, etc.
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x.
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks.
Ranked #21 on
Arithmetic Reasoning
on GSM8K
(using extra training data)
Mixture-of-Experts with Expert Choice Routing
Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning.
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Ranked #36 on
Code Generation
on HumanEval
Searching for Efficient Transformers for Language Modeling
For example, at a 500M parameter size, Primer improves the original T5 architecture on C4 auto-regressive language modeling, reducing the training cost by 4X.
Program Synthesis with Large Language Models
Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59. 6 percent of the problems from MBPP using few-shot learning with a well-designed prompt.
A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators
The rapidly-changing deep learning landscape presents a unique opportunity for building inference accelerators optimized for specific datacenter-scale workloads.
Carbon Emissions and Large Neural Network Training
To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal.
Ranked #1 on
Speech Recognition
on CHiME-6 eval
Searching for Fast Model Families on Datacenter Accelerators
On top of our DC accelerator optimized neural architecture search space, we further propose a latency-aware compound scaling (LACS), the first multi-objective compound scaling method optimizing both accuracy and latency.
Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour
EfficientNets are a family of state-of-the-art image classification models based on efficiently scaled convolutional neural networks.
Efficient Scale-Permuted Backbone with Learned Resource Distribution
Furthermore, SpineNet is built with a uniform resource distribution over operations.
Can weight sharing outperform random architecture search? An investigation with TuNAS
Efficient Neural Architecture Search methods based on weight sharing have shown good promise in democratizing Neural Architecture Search for computer vision models.
Go Wide, Then Narrow: Efficient Training of Deep Thin Networks
This is achieved by layerwise imitation, that is, forcing the thin network to mimic the intermediate outputs of the wide network from layer to layer.
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
Ranked #30 on
Neural Architecture Search
on ImageNet
Scaling Up Neural Architecture Search with Big Single-Stage Models
In this work, we propose BigNAS, an approach that simplifies this workflow and scales up neural architecture search to target a wide range of model sizes simultaneously.
Natural Questions: a Benchmark for Question Answering Research
The public release consists of 307, 373 training examples with single annotations, 7, 830 examples with 5-way annotations for development data, and a further 7, 842 examples 5-way annotated sequestered as test data.
Ranked #7 on
Question Answering
on Natural Questions (long)
Using Videos to Evaluate Image Model Robustness
Compared to previous studies on adversarial examples and synthetic distortions, natural robustness captures a more diverse set of common image transformations that occur in the natural environment.
AirDialogue: An Environment for Goal-Oriented Dialogue Research
However, current datasets are limited in size, and the environment for training agents and evaluating progress is relatively unsophisticated.
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate.
Efficient Neural Architecture Search via Parameters Sharing
We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design.
Ranked #31 on
Neural Architecture Search
on NAS-Bench-201, CIFAR-10
Understanding and Simplifying One-Shot Architecture Search
There is growing interest in automating neural network architecture design.
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge.
Scalable and accurate deep learning for electronic health records
Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality.
Unsupervised Pretraining for Sequence to Sequence Learning
We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models.
Tacotron: Towards End-to-End Speech Synthesis
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.
Ranked #5 on
Speech Synthesis
on North American English
Massive Exploration of Neural Machine Translation Architectures
Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users.
Large-Scale Evolution of Image Classifiers
Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone.
Ranked #120 on
Image Classification
on CIFAR-10
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters.
Ranked #14 on
Language Modelling
on One Billion Word
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (Short Version)
In this work, we propose the Manager-Programmer-Computer framework, which integrates neural networks with non-differentiable memory to support abstract, scalable and precise operations through a friendly neural computer interface.
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Harnessing the statistical power of neural networks to perform language understanding and symbolic reasoning is difficult, when it requires executing efficient discrete operations against a large knowledge-base.
A Neural Conversational Model
We find that this straightforward model can generate simple conversations given a large conversational training dataset.
Using Web Co-occurrence Statistics for Improving Image Categorization
Albeit the simplicity of the resulting optimization problem, it is effective in improving both recognition and localization accuracy.
