Search Results for author:
Found 43 papers, 26 papers with code
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations.
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022).
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Coreference Resolution
on WSC
Cross-Lingual Question Answering
Multi-task Language Understanding
+1
A Review of Sparse Expert Models in Deep Learning
Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Natural Language Inference
on RTE
ST-MoE: Designing Stable and Transferable Sparse Expert Models
But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
Ranked #1 on
Common Sense Reasoning
on ARC (Easy)
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing.
Ranked #9 on
Question Answering
on TriviaQA
(using extra training data)
Multi-Task Self-Training for Learning General Representations
The results suggest self-training is a promising direction to aggregate labeled and unlabeled training data for learning general feature representations.
Simple Training Strategies and Model Scaling for Object Detection
We benchmark these improvements on the vanilla ResNet-FPN backbone with RetinaNet and RCNN detectors.
Ranked #53 on
Object Detection
on COCO minival
Revisiting ResNets: Improved Training and Scaling Strategies
Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1. 7x - 2. 7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet.
Ranked #1 on
Document Image Classification
on AIP
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources.
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3. 6 mask AP on rare categories.
Ranked #1 on
Object Detection
on PASCAL VOC 2007
(using extra training data)
Does Data Augmentation Benefit from Split BatchNorms
Data augmentation has emerged as a powerful technique for improving the performance of deep neural networks and led to state-of-the-art results in computer vision.
Rethinking Pre-training and Self-training
For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data.
Ranked #1 on
Semantic Segmentation
on PASCAL VOC 2012 val
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation
We view this work as a notable step towards building a simple procedure to harness unlabeled video sequences and extra images to surpass state-of-the-art performance on core computer vision tasks.
Improving 3D Object Detection through Progressive Population Based Augmentation
Data augmentation has been widely adopted for object detection in 3D point clouds.
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
RandAugment: Practical automated data augmentation with a reduced search space
Additionally, due to the separate search phase, these approaches are unable to adjust the regularization strength based on model or dataset size.
Ranked #12 on
Data Augmentation
on ImageNet
Learning Data Augmentation Strategies for Object Detection
Importantly, the best policy found on COCO may be transferred unchanged to other detection datasets and models to improve predictive accuracy.
Ranked #83 on
Object Detection
on COCO test-dev
Attention Augmented Convolutional Networks
Convolutional networks have been the paradigm of choice in many computer vision applications.
Ranked #114 on
Image Classification
on CIFAR-100
(using extra training data)
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
On LibriSpeech, we achieve 6. 8% WER on test-other without the use of a language model, and 5. 8% WER with shallow fusion with a language model.
Ranked #1 on
Speech Recognition
on Hub5'00 SwitchBoard
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks.
Ranked #1 on
Human Part Segmentation
on PASCAL-Person-Part
Efficient Neural Architecture Search via Parameters Sharing
We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design.
Ranked #31 on
Neural Architecture Search
on NAS-Bench-201, CIFAR-10
Understanding and Simplifying One-Shot Architecture Search
There is growing interest in automating neural network architecture design.
AutoAugment: Learning Augmentation Policies from Data
In our implementation, we have designed a search space where a policy consists of many sub-policies, one of which is randomly chosen for each image in each mini-batch.
Ranked #6 on
Data Augmentation
on ImageNet
Efficient Neural Architecture Search via Parameter Sharing
The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set.
EXPLORING NEURAL ARCHITECTURE SEARCH FOR LANGUAGE TASKS
Neural architecture search (NAS), the task of finding neural architectures automatically, has recently emerged as a promising approach for unveiling better models over human-designed ones.
Faster Discovery of Neural Architectures by Searching for Paths in a Large Model
We propose Efficient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods.
Progressive Neural Architecture Search
We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms.
Ranked #14 on
Neural Architecture Search
on NAS-Bench-201, ImageNet-16-120
(Accuracy (Val) metric)
Intriguing Properties of Adversarial Examples
Finally, we study the effect of network architectures on adversarial sensitivity.
Searching for Activation Functions
The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.
Neural Optimizer Search with Reinforcement Learning
We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures.
Neural Optimizer Search using Reinforcement Learning
We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures.
Learning Transferable Architectures for Scalable Image Recognition
In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture".
Ranked #6 on
Classification
on InDL
Neural Architecture Search with Reinforcement Learning
Our cell achieves a test set perplexity of 62. 4 on the Penn Treebank, which is 3. 6 perplexity better than the previous state-of-the-art model.
Transfer Learning for Low-Resource Neural Machine Translation
Ensembling and unknown word replacement add another 2 Bleu which brings the NMT performance on low-resource machine translation close to a strong syntax based machine translation (SBMT) system, exceeding its performance on one language pair.
Multi-Source Neural Translation
We build a multi-source machine translation model and train it to maximize the probability of a target English string given French and German sources.
