Search Results for author:
Found 130 papers, 83 papers with code
Symbolic Discovery of Optimization Algorithms
We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training.
Unified Functional Hashing in Automatic Machine Learning
In this paper, we show that large efficiency gains can be obtained by employing a fast unified functional hash, especially through the functional equivalence caching technique, which we also present.
Noise2Music: Text-conditioned Music Generation with Diffusion Models
We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts.
PyGlove: Efficiently Exchanging ML Ideas as Code
We also perform a case study of a large codebase where PyGlove led to an 80% reduction in the number of lines of code.
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022).
Inverse scaling can become U-shaped
The Inverse Scaling Prize (McKenzie et al. 2022) identified eleven such inverse scaling tasks, evaluated on models of up to 280B parameters and up to 500 zettaFLOPs of training compute.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Multi-task Language Understanding
on BBH-nlp
Cross-Lingual Question Answering
Multi-task Language Understanding
+1
Transcending Scaling Laws with 0.1% Extra Compute
This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute.
Ranked #1 on
Question Answering
on StrategyQA
G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR
Data augmentation is a ubiquitous technique used to provide robustness to automatic speech recognition (ASR) training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models.
Revisiting Multi-Scale Feature Fusion for Semantic Segmentation
It is commonly believed that high internal resolution combined with expensive operations (e. g. atrous convolutions) are necessary for accurate semantic segmentation, resulting in slow speed and large memory usage.
DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection
In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e. g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion.
Transformer Quality in Linear Time
We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences.
Ranked #1 on
Language Modelling
on Wiki-40B
Combined Scaling for Open-Vocabulary Image Classification
Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood.
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Primer: Searching for Efficient Transformers for Language Modeling
For example, at a 500M parameter size, Primer improves the original T5 architecture on C4 auto-regressive language modeling, reducing the training cost by 4X.
Ranked #1 on
Language Modelling
on C4
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available.
Ranked #1 on
Few-Shot NLI
on SNLI (8 training examples per class)
Finetuned Language Models Are Zero-Shot Learners
We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks.
Ranked #1 on
Question Answering
on StoryCloze
Multi-Task Self-Training for Learning General Representations
The results suggest self-training is a promising direction to aggregate labeled and unlabeled training data for learning general feature representations.
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks.
Ranked #1 on
Image Classification
on GasHisSDB
Pay Attention to MLPs
Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years.
Ranked #22 on
Natural Language Inference
on MultiNLI
EfficientNetV2: Smaller Models and Faster Training
By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87. 3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2. 0% accuracy while training 5x-11x faster using the same computing resources.
Ranked #2 on
Image Classification
on Stanford Cars
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset.
Ranked #1 on
Image Classification
on VTAB-1k
(using extra training data)
PyGlove: Symbolic Programming for Automated Machine Learning
As a result, AutoML can be reformulated as an automated process of symbolic manipulation.
Evolving Reinforcement Learning Algorithms
Learning from scratch on simple classical control and gridworld tasks, our method rediscovers the temporal-difference (TD) algorithm.
AutoDropout: Learning Dropout Patterns to Regularize Deep Networks
As a result, these conventional methods are less effective than methods that leverage the structures, such as SpatialDropout and DropBlock, which randomly drop the values at certain contiguous areas in the hidden states and setting them to zero.
Ranked #1 on
Image Classification
on cifar-10,4000
Pre-Training Transformers as Energy-Based Cloze Models
We introduce Electric, an energy-based cloze model for representation learning over text.
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3. 6 mask AP on rare categories.
Ranked #2 on
Object Detection
on PASCAL VOC 2007
Towards Domain-Agnostic Contrastive Learning
Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation.
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset.
Ranked #1 on
Speech Recognition
on LibriSpeech test-clean
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Smooth Adversarial Training
SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82. 2% accuracy and 58. 6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9. 5% for accuracy and 11. 6% for robustness.
Rethinking Pre-training and Self-training
For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data.
Ranked #1 on
Semantic Segmentation
on PASCAL VOC 2012 val
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost.
Ranked #6 on
Reading Comprehension
on RACE
AutoHAS: Efficient Hyperparameter and Architecture Search
Efficient hyperparameter or architecture search methods have shown remarkable results, but each of them is only applicable to searching for either hyperparameters (HPs) or architectures.
Improved Noisy Student Training for Automatic Speech Recognition
Noisy student training is an iterative self-training method that leverages augmentation to improve network performance.
Ranked #4 on
Speech Recognition
on LibriSpeech test-clean
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension
Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning.
Ranked #4 on
Question Answering
on DROP Test
Chip Placement with Deep Reinforcement Learning
To achieve these results, we pose placement as a Reinforcement Learning (RL) problem and train an agent to place the nodes of a chip netlist onto a chip canvas.
Evolving Normalization-Activation Layers
Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other.
Improving 3D Object Detection through Progressive Population Based Augmentation
Data augmentation has been widely adopted for object detection in 3D point clouds.
Meta Pseudo Labels
We present Meta Pseudo Labels, a semi-supervised learning method that achieves a new state-of-the-art top-1 accuracy of 90. 2% on ImageNet, which is 1. 6% better than the existing state-of-the-art.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not.
Ranked #7 on
Question Answering
on Quora Question Pairs
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces.
Towards a Human-like Open-Domain Chatbot
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations.
SpecAugment on Large Scale Datasets
Recently, SpecAugment, an augmentation scheme for automatic speech recognition that acts directly on the spectrogram of input utterances, has shown to be highly effective in enhancing the performance of end-to-end networks on public datasets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search.
Ranked #8 on
Image Classification
on iNaturalist
MnasFPN: Learning Latency-aware Pyramid Architecture for Object Detection on Mobile Devices
We propose MnasFPN, a mobile-friendly search space for the detection head, and combine it with latency-aware architecture search to produce efficient object detection models.
Ranked #231 on
Object Detection
on COCO test-dev
Mixtape: Breaking the Softmax Bottleneck Efficiently
The softmax bottleneck has been shown to limit the expressiveness of neural language models.
Adversarial Examples Improve Image Recognition
We show that AdvProp improves a wide range of models on various image recognition tasks and performs better when the models are bigger.
Ranked #186 on
Image Classification
on ImageNet
EfficientDet: Scalable and Efficient Object Detection
Model efficiency has become increasingly important in computer vision.
Ranked #6 on
Object Detection
on COCO minival
(APS metric)
Self-training with Noisy Student improves ImageNet classification
During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment to the student so that the student generalizes better than the teacher.
Ranked #14 on
Image Classification
on ImageNet ReaL
(using extra training data)
High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
Predicting future video frames is extremely challenging, as there are many factors of variation that make up the dynamics of how frames change through time.
RandAugment: Practical automated data augmentation with a reduced search space
Additionally, due to the separate search phase, these approaches are unable to adjust the regularization strength based on model or dataset size.
Ranked #10 on
Data Augmentation
on ImageNet
Semi-supervised Learning by Coaching
Recent semi-supervised learning (SSL) methods often have a teacher to train a student in order to propagate labels from labeled data to unlabeled data.
Saccader: Improving Accuracy of Hard Attention Models for Vision
Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret.
MixConv: Mixed Depthwise Convolutional Kernels
In this paper, we systematically study the impact of different kernel sizes, and observe that combining the benefits of multiple kernel sizes can lead to better accuracy and efficiency.
Ranked #646 on
Image Classification
on ImageNet
Neural Input Search for Large Scale Recommendation Models
During training we use reinforcement learning to find the optimal vocabulary size for each feature and embedding dimension for each value of the feature.
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts.
Learning Data Augmentation Strategies for Object Detection
Importantly, the best policy found on COCO may be transferred unchanged to other detection datasets and models to improve predictive accuracy.
Ranked #80 on
Object Detection
on COCO test-dev
XLNet: Generalized Autoregressive Pretraining for Language Understanding
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling.
Selfie: Self-supervised Pretraining for Image Embedding
Notably, on ImageNet 224 x 224 with 60 examples per class (5%), our method improves the mean accuracy of ResNet-50 from 35. 6% to 46. 7%, an improvement of 11. 1 points in absolute accuracy.
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available.
Ranked #1 on
Medical Image Classification
on NCT-CRC-HE-100K
Fine-Grained Image Classification
Medical Image Classification
+2
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
We find that the optimal SGD hyper-parameters are determined by a "normalized noise scale," which is a function of the batch size, learning rate, and initialization conditions.
Searching for MobileNetV3
We achieve new state of the art results for mobile classification, detection and segmentation.
Ranked #4 on
Dichotomous Image Segmentation
on DIS-TE3
Transformer-XL: Language Modeling with Longer-Term Dependency
Moreover, Transformer-XL is up to 1, 800+ times faster than vanilla Transformer during evaluation.
Diversity and Depth in Per-Example Routing Models
Both architectural diversity and routing depth can increase the representational power of a routing network.
Ranked #2 on
Multi-Task Learning
on OMNIGLOT
Do Language Models Have Common Sense?
It has been argued that current machine learning models do not have commonsense, and therefore must be hard-coded with prior knowledge (Marcus, 2018).
Unsupervised Data Augmentation for Consistency Training
In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.
Ranked #1 on
Sentiment Analysis
on Amazon Review Full
Attention Augmented Convolutional Networks
Convolutional networks have been the paradigm of choice in many computer vision applications.
Ranked #109 on
Image Classification
on CIFAR-100
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
On LibriSpeech, we achieve 6. 8% WER on test-other without the use of a language model, and 5. 8% WER with shallow fusion with a language model.
Ranked #1 on
Speech Recognition
on Hub5'00 SwitchBoard
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection
Here we aim to learn a better architecture of feature pyramid network for object detection.
CondConv: Conditionally Parameterized Convolutions for Efficient Inference
We demonstrate that scaling networks with CondConv improves the performance and inference cost trade-off of several existing convolutional neural network architectures on both classification and detection tasks.
Ranked #682 on
Image Classification
on ImageNet
The Evolved Transformer
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models.
Ranked #1 on
Machine Translation
on WMT2014 English-Czech
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.
Ranked #3 on
Language Modelling
on One Billion Word
Domain Adaptive Transfer Learning with Specialist Models
Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning.
Ranked #2 on
Fine-Grained Image Classification
on Stanford Cars
(using extra training data)
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks.
Ranked #4 on
Fine-Grained Image Classification
on Birdsnap
(using extra training data)
DropBlock: A regularization method for convolutional networks
This lack of success of dropout for convolutional layers is perhaps due to the fact that activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout.
Ranked #681 on
Image Classification
on ImageNet
Semi-Supervised Sequence Modeling with Cross-View Training
We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data.
Ranked #3 on
CCG Supertagging
on CCGbank
MnasNet: Platform-Aware Neural Architecture Search for Mobile
In this paper, we propose an automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency.
Ranked #731 on
Image Classification
on ImageNet
Stochastic natural gradient descent draws posterior samples in function space
Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima.
A Simple Method for Commonsense Reasoning
Commonsense reasoning is a long-standing challenge for deep learning.
Ranked #6 on
Common Sense Reasoning
on Winograd Schema Challenge
AutoAugment: Learning Augmentation Policies from Data
In our implementation, we have designed a search space where a policy consists of many sub-policies, one of which is randomly chosen for each image in each mini-batch.
Ranked #4 on
Data Augmentation
on ImageNet
Do Better ImageNet Models Transfer Better?
Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer.
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models.
Ranked #27 on
Question Answering
on SQuAD1.1 dev
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge.
Ranked #10 on
Sequential Image Classification
on Sequential CIFAR-10
Efficient Neural Architecture Search via Parameter Sharing
The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set.
Regularized Evolution for Image Classifier Architecture Search
The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically.
Cross-View Training for Semi-Supervised Learning
The students can learn from the teacher (the full model) because the teacher sees more of each example.
Ranked #4 on
Chunking
on CoNLL 2000
(using extra training data)
Faster Discovery of Neural Architectures by Searching for Paths in a Large Model
We propose Efficient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods.
Code Synthesis with Priority Queue Training
We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards.
A Hierarchical Model for Device Placement
We introduce a hierarchical model for efficient placement of computational graphs onto hardware devices, especially in heterogeneous environments with a mixture of CPUs, GPUs, and other computational devices.
A Goal-oriented Neural Conversation Model by Self-Play
One challenge in applying such techniques to building goal-oriented conversation models is that maximum likelihood-based models are not optimized toward accomplishing goals.
EXPLORING NEURAL ARCHITECTURE SEARCH FOR LANGUAGE TASKS
Neural architecture search (NAS), the task of finding neural architectures automatically, has recently emerged as a promising approach for unveiling better models over human-designed ones.
Intriguing Properties of Adversarial Examples
Finally, we study the effect of network architectures on adversarial sensitivity.
Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?
Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization.
Don't Decay the Learning Rate, Increase the Batch Size
We can further reduce the number of parameter updates by increasing the learning rate $\epsilon$ and scaling the batch size $B \propto \epsilon$.
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
Interpreting stochastic gradient descent as a stochastic differential equation, we identify the "noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$ the batch size.
Searching for Activation Functions
The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.
Neural Optimizer Search with Reinforcement Learning
We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures.
Neural Optimizer Search using Reinforcement Learning
We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures.
Learning Transferable Architectures for Scalable Image Recognition
In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture".
Ranked #31 on
Image Classification
on ImageNet ReaL
Device Placement Optimization with Reinforcement Learning
Key to our method is the use of a sequence-to-sequence model to predict which subsets of operations in a TensorFlow graph should run on which of the available devices.
Learning to Skim Text
Recurrent Neural Networks are showing much promise in many sub-areas of natural language processing, ranging from document classification to machine translation to automatic question answering.
An Online Sequence-to-Sequence Model Using Partial Conditioning
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Neural Combinatorial Optimization with Reinforcement Learning
Despite the computational expense, without much engineering and heuristic designing, Neural Combinatorial Optimization achieves close to optimal results on 2D Euclidean graphs with up to 100 nodes.
Learning a Natural Language Interface with Neural Programmer
The main experimental result in this paper is that a single Neural Programmer model achieves 34. 2% accuracy using only 10, 000 examples with weak supervision.
Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation.
Unsupervised Pretraining for Sequence to Sequence Learning
We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models.
Neural Architecture Search with Reinforcement Learning
Our cell achieves a test set perplexity of 62. 4 on the Penn Treebank, which is 3. 6 perplexity better than the previous state-of-the-art model.
HyperNetworks
This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network.
Ranked #14 on
Language Modelling
on Penn Treebank (Character Level)
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder.
Ranked #35 on
Machine Translation
on WMT2014 English-French
Adding Gradient Noise Improves Learning for Very Deep Networks
This success is partially attributed to architectural innovations such as convolutional and long short-term memory networks.
Multi-task Sequence to Sequence Learning
This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation.
Neural Programmer: Inducing Latent Programs with Gradient Descent
In this work, we propose Neural Programmer, an end-to-end differentiable neural network augmented with a small set of basic arithmetic and logic operations.
A Neural Transducer
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Semi-supervised Sequence Learning
In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better.
Listen, Attend and Spell
Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly.
Document Embedding with Paragraph Vectors
Paragraph Vectors has been recently proposed as an unsupervised method for learning distributed representations for pieces of texts.
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients.
Ranked #25 on
Sequential Image Classification
on Sequential MNIST
Addressing the Rare Word Problem in Neural Machine Translation
Our experiments on the WMT14 English to French translation task show that this method provides a substantial improvement of up to 2. 8 BLEU points over an equivalent NMT system that does not use this technique.
Ranked #40 on
Machine Translation
on WMT2014 English-French
Sequence to Sequence Learning with Neural Networks
Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector.
Ranked #5 on
Traffic Prediction
on PeMS-M
(using extra training data)
Distributed Representations of Sentences and Documents
Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models.
Ranked #4 on
Question Answering
on QASent
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Previous work on Recursive Neural Networks (RNNs) shows that these models can produce compositional feature vectors for accurately representing and classifying sentences or images.
Exploiting Similarities among Languages for Machine Translation
Dictionaries and phrase tables are the basis of modern statistical machine translation systems.
Large Scale Distributed Deep Networks
Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance.
Building high-level features using large scale unsupervised learning
For example, is it possible to learn a face detector using only unlabeled images?
ICA with Reconstruction Cost for Efficient Overcomplete Feature Learning
We show that the soft reconstruction cost can also be used to prevent replicated features in tiled convolutional neural networks.
Ranked #116 on
Image Classification
on STL-10
Tiled convolutional neural networks
Using convolutional (tied) weights significantly reduces the number of parameters that have to be learned, and also allows translational invariance to be hard-coded into the architecture.
Measuring Invariances in Deep Networks
Our evaluation metrics can also be used to evaluate future work in unsupervised deep learning, and thus help the development of future algorithms.
Tighter Bounds for Structured Estimation
Large-margin structured estimation methods work by minimizing a convex upper bound of loss functions.
