Search Results for author:
Found 576 papers, 295 papers with code
Learning the 2-D Topology of Images
We study the following question: is the two-dimensional structure of images a very strong prior or is it something that can be learned with a few examples of natural images?
Topmoumoute Online Natural Gradient Algorithm
Guided by the goal of obtaining an optimization algorithm that is both fast and yielding good generalization, we study the descent direction maximizing the decrease in generalization error or the probability of not increasing generalization error.
Slow, Decorrelated Features for Pretraining Complex Cell-like Networks
We introduce a new type of neural network activation function based on recent physiological rate models for complex cells in visual area V1.
An Infinite Factor Model Hierarchy Via a Noisy-Or Mechanism
The Indian Buffet Process is a Bayesian nonparametric approach that models objects as arising from an infinite number of latent factors.
Understanding the difficulty of training deep feedforward neural networks
Whereas before 2006 it appears that deep multi-layer neural networks were not successfully trained, since then several algorithms have been shown to successfully train them, with experimental results showing the superiority of deeper vs less deep architectures.
Adaptive Drift-Diffusion Process to Learn Time Intervals
We provide an analytical proof that the model can learn inter-event intervals in a number of trials independent of the interval size and that the temporal precision of the system is proportional to the timed interval.
Algorithms for Hyper-Parameter Optimization
Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreliable for training DBNs.
On Tracking The Partition Function
In this paper, we exploit the gradient descent training procedure of restricted Boltzmann machines (a type of MRF) to {\bf track} the log partition function during learning.
Shallow vs. Deep Sum-Product Networks
We investigate the representational power of sum-product networks (computation networks analogous to neural networks, but whose individual units compute either products or weighted sums), through a theoretical analysis that compares deep (multiple hidden layers) vs. shallow (one hidden layer) architectures.
Representation Learning: A Review and New Perspectives
The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data.
Practical recommendations for gradient-based training of deep architectures
Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters.
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding
We consider the problem of object recognition with a large number of classes.
Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
We investigate the problem of modeling symbolic sequences of polyphonic music in a completely general piano-roll representation.
Ranked #5 on
Music Modeling
on JSB Chorales
Better Mixing via Deep Representations
It has previously been hypothesized, and supported with some experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation.
Efficient EM Training of Gaussian Mixtures with Missing Data
In data-mining applications, we are frequently faced with a large fraction of missing entries in the data matrix, which is problematic for most discriminant machine learning algorithms.
Disentangling Factors of Variation via Generative Entangling
Seen from a generative perspective, the multiplicative interactions emulates the entangling of factors of variation.
What Regularized Auto-Encoders Learn from the Data Generating Distribution
This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density.
On the difficulty of training Recurrent Neural Networks
There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994).
Theano: new features and speed improvements
Theano is a linear algebra compiler that optimizes a user's symbolically-specified mathematical computations to produce efficient low-level implementations.
Advances in Optimizing Recurrent Networks
After a more than decade-long period of relatively little research activity in the area of recurrent neural networks, several new developments will be reviewed here that have allowed substantial progress both in understanding and in technical solutions towards more efficient training of recurrent networks.
A Semantic Matching Energy Function for Learning with Multi-relational Data
Large-scale relational learning becomes crucial for handling the huge amounts of structured data generated daily in many application domains ranging from computational biology or information retrieval, to natural language processing.
Joint Training Deep Boltzmann Machines for Classification
We introduce a new method for training deep Boltzmann machines jointly.
Revisiting Natural Gradient for Deep Networks
We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models.
Knowledge Matters: Importance of Prior Information for Optimization
We explore the effect of introducing prior information into the intermediate level of neural networks for a learning task on which all the state-of-the-art machine learning algorithms tested failed to learn.
Maxout Networks
We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout.
Ranked #34 on
Image Classification
on MNIST
Deep Learning of Representations: Looking Forward
Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts.
Estimating or Propagating Gradients Through Stochastic Neurons
The second approach we propose assumes that an estimator of the gradient can be back-propagated and it provides an unbiased estimator of the gradient, but can only work with non-linearities unlike the hard threshold, but like the rectifier, that are not flat for all of their range.
Generalized Denoising Auto-Encoders as Generative Models
Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued.
Deep Generative Stochastic Networks Trainable by Backprop
We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood.
Challenges in Representation Learning: A report on three machine learning contests
The ICML 2013 Workshop on Challenges in Representation Learning focused on three challenges: the black box learning challenge, the facial expression recognition challenge, and the multimodal learning challenge.
Ranked #12 on
Facial Expression Recognition (FER)
on FER2013
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons?
Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
In this paper we propose and investigate a novel nonlinear unit, called $L_p$ unit, for deep neural networks.
Bounding the Test Log-Likelihood of Generative Models
Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable normalization.
Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs
Sparse high-dimensional data vectors are common in many application domains where a very large number of rarely non-zero features can be devised.
Multi-Prediction Deep Boltzmann Machines
We introduce the Multi-Prediction Deep Boltzmann Machine (MP-DBM).
On the Challenges of Physical Implementations of RBMs
Restricted Boltzmann machines (RBMs) are powerful machine learning models, but learning and some kinds of inference in the model require sampling-based approximations, which, in classical digital computers, are implemented using expensive MCMC.
Multimodal Transitions for Generative Stochastic Networks
Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distribution.
On the number of response regions of deep feed forward networks with piece-wise linear activations
For a $k$ layer model with $n$ hidden units on each layer it is $\Omega(\left\lfloor {n}/{n_0}\right\rfloor^{k-1}n^{n_0})$.
How to Construct Deep Recurrent Neural Networks
Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996).
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Catastrophic forgetting is a problem faced by many machine learning models and algorithms.
An empirical analysis of dropout in piecewise linear networks
The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters.
On the Number of Linear Regions of Deep Neural Networks
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have.
On the saddle point problem for non-convex optimization
Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for the ability of these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum.
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN).
Ranked #47 on
Machine Translation
on WMT2014 English-French
Iterative Neural Autoregressive Distribution Estimator (NADE-k)
Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data.
Ranked #7 on
Image Generation
on Binarized MNIST
Generative Adversarial Networks
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake.
Super-Resolution
Time-Series Few-Shot Learning with Heterogeneous Channels
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum.
Reweighted Wake-Sleep
The wake-sleep algorithm relies on training not just the directed generative model but also a conditional generative model (the inference network) that runs backward from visible to latent, estimating the posterior distribution of latent given visible.
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation required, by activating some parameters and computation "on-demand", on a per-example basis.
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
We propose to exploit {\em reconstruction} as a layer-local training signal for deep learning.
Neural Machine Translation by Jointly Learning to Align and Translate
Neural machine translation is a recently proposed approach to machine translation.
Ranked #4 on
Dialogue Generation
on Persona-Chat
(using extra training data)
On the Equivalence Between Deep NADE and Generative Stochastic Networks
Orderless NADEs are trained based on a criterion that stochastically maximizes $P(\mathbf{x})$ with all possible orders of factorizations.
Overcoming the Curse of Sentence Length for Neural Machine Translation using Automatic Segmentation
The authors of (Cho et al., 2014a) have shown that the recently introduced neural network translation systems suffer from a significant drop in translation quality when translating long sentences, unlike existing phrase-based translation systems.
On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
In this paper, we focus on analyzing the properties of the neural machine translation using two models; RNN Encoder--Decoder and a newly proposed gated recursive convolutional neural network.
Deep Tempering
Restricted Boltzmann Machines (RBMs) are one of the fundamental building blocks of deep learning.
Deep Directed Generative Autoencoders
The objective is to learn an encoder $f(\cdot)$ that maps $X$ to $f(X)$ that has a much simpler distribution than $X$ itself, estimated by $P(H)$.
Not All Neural Embeddings are Born Equal
Neural language models learn word representations that capture rich linguistic and conceptual information.
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
We introduce BilBOWA (Bilingual Bag-of-Words without Alignments), a simple and computationally-efficient model for learning bilingual distributed representations of words which can scale to large monolingual datasets and does not require word-aligned parallel training data.
Ranked #1 on
Document Classification
on Reuters En-De
Cross-Lingual Document Classification
Document Classification
+3
NICE: Non-linear Independent Components Estimation
It is based on the idea that a good representation is one in which the data has a distribution that is easy to model.
Ranked #73 on
Image Generation
on CIFAR-10
(bits/dimension metric)
How transferable are features in deep neural networks?
Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks.
Iterative Neural Autoregressive Distribution Estimator NADE-k
Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data.
Ranked #8 on
Image Generation
on Binarized MNIST
Generative Adversarial Nets
We propose a new framework for estimating generative models via adversarial nets, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake.
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes.
On Using Very Large Target Vocabulary for Neural Machine Translation
The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models.
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
In this paper we compare different types of recurrent units in recurrent neural networks (RNNs).
Ranked #10 on
Music Modeling
on JSB Chorales
Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review).
FitNets: Hints for Thin Deep Nets
In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student.
Embedding Word Similarity with Neural Machine Translation
Here we investigate the embeddings learned by neural machine translation models, a recently-developed class of neural language model.
Training deep neural networks with low precision multiplications
For each of those datasets and for each of those formats, we assess the impact of the precision of the multiplications on the final error after training.
Difference Target Propagation
Back-propagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment.
ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients.
Gated Feedback Recurrent Neural Networks
In this work, we propose a novel recurrent neural network (RNN) architecture.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images.
Towards Biologically Plausible Deep Learning
Neuroscientists have long criticised deep learning algorithms as incompatible with current knowledge of neurobiology.
Equilibrated adaptive learning rates for non-convex optimization
Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks.
EmoNets: Multimodal deep learning approaches for emotion recognition in video
The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies.
On Using Monolingual Corpora in Neural Machine Translation
Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation.
GSNs : Generative Stochastic Networks
We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood.
Learning to Understand Phrases by Embedding the Dictionary
Distributional models that learn rich semantic word representations are a success story of recent NLP research.
ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
In this paper, we propose a deep neural network architecture for object recognition based on recurrent neural networks.
Ranked #34 on
Image Classification
on MNIST
Brain Tumor Segmentation with Deep Neural Networks
Finally, we explore a cascade architecture in which the output of a basic CNN is treated as an additional source of information for a subsequent CNN.
Ranked #1 on
Brain Tumor Segmentation
on BRATS-2013 leaderboard
Blocks and Fuel: Frameworks for deep learning
We introduce two Python frameworks to train neural networks on large datasets: Blocks and Fuel.
A Recurrent Latent Variable Model for Sequential Data
In this paper, we explore the inclusion of latent random variables into the dynamic hidden state of a recurrent neural network (RNN) by combining elements of the variational autoencoder.
Bidirectional Helmholtz Machines
We present a lower-bound for the likelihood of this model and we show that optimizing this bound regularizes the model so that the Bhattacharyya distance between the bottom-up and top-down approximate distributions is minimized.
Attention-Based Models for Speech Recognition
Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks in- cluding machine translation, handwriting synthesis and image caption gen- eration.
Ranked #17 on
Speech Recognition
on TIMIT
Describing Multimedia Content using Attention-based Encoder--Decoder Networks
Whereas deep neural networks were first mostly used for classification tasks, they are rapidly expanding in the realm of structured output problems, where the observed target is composed of multiple random variables that have a rich joint distribution, given the input.
A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
Our novel hierarchical recurrent encoder-decoder architecture allows the model to be sensitive to the order of queries in the context while avoiding data sparsity.
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
We investigate the task of building open domain, conversational dialogue systems based on large dialogue corpora using generative models.
Clustering is Efficient for Approximate Maximum Inner Product Search
Efficient Maximum Inner Product Search (MIPS) is an important task that has a wide applicability in recommendation systems and classification with a large number of classes.
Artificial Neural Networks Applied to Taxi Destination Prediction
We describe our first-place solution to the ECML/PKDD discovery challenge on taxi destination prediction.
End-to-End Attention-based Large Vocabulary Speech Recognition
Many of the current state-of-the-art Large Vocabulary Continuous Speech Recognition Systems (LVCSR) are hybrids of neural networks and Hidden Markov Models (HMMs).
STDP as presynaptic activity times rate of change of postsynaptic activity
We introduce a weight update formula that is expressed only in terms of firing rates and their derivatives and that results in changes consistent with those associated with spike-timing dependent plasticity (STDP) rules and biological observations, even though the explicit timing of spikes is not needed.
Batch Normalized Recurrent Neural Networks
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies.
Early Inference in Energy-Based Models Approximates Back-Propagation
We show that Langevin MCMC inference in an energy-based model with latent variables has the property that the early steps of inference, starting from a stationary point, correspond to propagating error gradients into internal layers, similarly to back-propagation.
Neural Networks with Few Multiplications
For most deep learning algorithms training is notoriously time consuming.
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated.
Ranked #30 on
Image Classification
on SVHN
Oracle performance for visual captioning
The task of associating images and videos with a natural language description has attracted a great amount of attention recently.
Deconstructing the Ladder Network Architecture
Although the empirical results are impressive, the Ladder Network has many components intertwined, whose contributions are not obvious in such a complex architecture.
Denoising Criterion for Variational Auto-Encoding Framework
Denoising autoencoders (DAE) are trained to reconstruct their clean inputs with noise injected at the input level, while variational autoencoders (VAE) are trained with noise injected in their stochastic hidden layer, with a regularizer that encourages this noise injection.
Task Loss Estimation for Sequence Prediction
Our idea is that this score can be interpreted as an estimate of the task loss, and that the estimation error may be used as a consistent surrogate loss.
Unitary Evolution Recurrent Neural Networks
When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies.
Ranked #26 on
Sequential Image Classification
on Sequential MNIST
Variance Reduction in SGD by Distributed Importance Sampling
This leads the model to update using an unbiased estimate of the gradient which also has minimum variance when the sampling proposal is proportional to the L2-norm of the gradient.
ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation
Moreover, ReNet layers are stacked on top of pre-trained convolutional layers, benefiting from generic local features.
Ranked #18 on
Semantic Segmentation
on CamVid
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
We propose multi-way, multilingual neural machine translation.
Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time.
Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation
Because the objective function is defined in terms of local perturbations, the second phase of Equilibrium Propagation corresponds to only nudging the prediction (fixed point, or stationary distribution) towards a configuration that reduces prediction error.
Architectural Complexity Measures of Recurrent Neural Networks
In this paper, we systematically analyze the connecting architectures of recurrent neural networks (RNNs).
Ranked #23 on
Language Modelling
on Text8
Noisy Activation Functions
Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only).
A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation
The existing machine translation systems, whether phrase-based or neural, have relied almost exclusively on word-level modelling with explicit segmentation.
Ranked #3 on
Machine Translation
on WMT2015 English-German
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Over the past decade, large-scale supervised learning corpora have enabled machine learning researchers to make substantial advances.
Pointing the Unknown Words
At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.
Theano: A Python framework for fast computation of mathematical expressions
Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements.
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Sequential data often possesses a hierarchical structure with complex dependencies between subsequences, such as found between the utterances in a dialogue.
Hierarchical Memory Networks
In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks.
Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation
We introduce the multiresolution recurrent neural network, which extends the sequence-to-sequence framework to model natural language generation as two parallel discrete stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language tokens.
Ranked #1 on
Dialogue Generation
on Ubuntu Dialogue (Activity)
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
We propose zoneout, a novel method for regularizing RNNs.
Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible
We find conditions under which a simple feedforward computation is a very good initialization for inference, after the input units are clamped to observed values.
Iterative Alternating Neural Attention for Machine Reading
We propose a novel neural attention architecture to tackle machine comprehension tasks, such as answering Cloze-style queries with respect to a document.
Ranked #3 on
Question Answering
on Children's Book Test
(Accuracy-NE metric)
Deep Directed Generative Models with Energy-Based Probability Estimation
Training energy-based probabilistic models is confronted with apparently intractable sums, whose Monte Carlo estimation requires sampling from the estimated probability distribution in the inner loop of training.
Online and Offline Handwritten Chinese Character Recognition: A Comprehensive Study and New Benchmark
Furthermore, although directMap+convNet can achieve the best results and surpass human-level performance, we show that writer adaptation in this case is still effective.
Data Augmentation
Offline Handwritten Chinese Character Recognition
Drawing and Recognizing Chinese Characters with Recurrent Neural Network
In this paper, we propose a framework by using the recurrent neural network (RNN) as both a discriminative model for recognizing Chinese characters and a generative model for drawing (generating) Chinese characters.
On Multiplicative Integration with Recurrent Neural Networks
We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs).
Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
We investigate the mechanisms and effects of learning to read and write into a memory through experiments on Facebook bAbI tasks using both a feedforward and GRUcontroller.
Ranked #5 on
Question Answering
on bAbi
Context-Dependent Word Representation for Neural Machine Translation
Based on this observation, in this paper we propose to contextualize the word embedding vectors using a nonlinear bag-of-words representation of the source sentence.
HeMIS: Hetero-Modal Image Segmentation
We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities.
Ranked #98 on
Semantic Segmentation
on NYU Depth v2
An Actor-Critic Algorithm for Sequence Prediction
We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL).
Ranked #8 on
Machine Translation
on IWSLT2015 English-German
A Neural Knowledge Language Model
Current language models have a significant limitation in the ability to encode and decode factual knowledge.
Mollifying Networks
The optimization of deep neural networks can be more challenging than traditional convex optimization problems due to the highly non-convex nature of the loss function, e. g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape for algorithms based on simple gradient descent.
Recurrent Neural Networks With Limited Numerical Precision
We present results from the use of different stochastic and deterministic reduced precision training methods applied to three major RNN types which are then tested on several datasets.
Hierarchical Multiscale Recurrent Neural Networks
Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence.
Ranked #19 on
Language Modelling
on Text8
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits.
Understanding intermediate layers using linear classifier probes
Neural network models have a reputation for being black boxes.
Professor Forcing: A New Algorithm for Training Recurrent Networks
We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of the recurrent network to be the same when training the network and when sampling from the network over multiple time steps.
Recurrent Neural Networks With Limited Numerical Precision
Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high.
Invariant Representations for Noisy Speech Recognition
Ensuring such robustness to variability is a challenge in modern day neural network-based ASR systems, especially when all types of variability are not seen during training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
State-of-the-art approaches for semantic image segmentation are built on Convolutional Neural Networks (CNNs).
Ranked #9 on
Semantic Segmentation
on CamVid
Diet Networks: Thin Parameters for Fat Genomics
It is based on the idea that we can first learn or provide a distributed representation for each input feature (e. g. for each position in the genome where variations are observed), and then learn (with another neural network called the parameter prediction network) how to map a feature's distributed representation to the vector of parameters specific to that feature in the classifier neural network (the weights which link the value of the feature to each of the hidden units).
Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
PPGNs are composed of 1) a generator network G that is capable of drawing a wide range of image types and 2) a replaceable "condition" network C that tells the generator what to draw.
Mode Regularized Generative Adversarial Networks
Although Generative Adversarial Networks achieve state-of-the-art results on a variety of generative tasks, they are regarded as highly unstable and prone to miss modes.
Generalizable Features From Unsupervised Learning
In this work, we explore the potential of unsupervised learning to find features that promote better generalization to settings outside the supervised training distribution.
On Random Weights for Texture Generation in One Layer Neural Networks
Recent work in the literature has shown experimentally that one can use the lower layers of a trained convolutional neural network (CNN) to model natural textures.
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time.
Memory Augmented Neural Networks with Wormhole Connections
We use discrete addressing for read/write operations which helps to substantially to reduce the vanishing gradient problem with very long sequences.
Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation
Moreover, when applying our 2D pipeline on a challenging 3D MRI prostate segmentation challenge we reach results that are competitive even when compared to 3D methods.
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
Despite the successes in capturing continuous distributions, the application of generative adversarial networks (GANs) to discrete settings, like natural language tasks, is rather restricted.
Boundary-Seeking Generative Adversarial Networks
We introduce a method for training GANs with discrete data that uses the estimated difference measure from the discriminator to compute importance weights for generated samples, thus providing a policy gradient for training the generator.
A Robust Adaptive Stochastic Gradient Method for Deep Learning
The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients.
A Structured Self-attentive Sentence Embedding
This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention.
Sharp Minima Can Generalize For Deep Nets
Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice.
Independently Controllable Features
Finding features that disentangle the different causes of variation in real data is a difficult task, that has nonetheless received considerable attention in static domains like natural images.
A network of deep neural networks for distant speech recognition
Despite the remarkable progress recently made in distant speech recognition, state-of-the-art technology still suffers from a lack of robustness, especially when adverse acoustic conditions characterized by non-stationary noises and reverberation are met.
Batch-normalized joint training for DNN-based distant speech recognition
Improving distant speech recognition is a crucial step towards flexible human-machine interfaces.
Count-ception: Counting by Fully Convolutional Redundant Counting
Our contribution is redundant counting instead of predicting a density map in order to average over errors.
Image Segmentation by Iterative Inference from Conditional Score Estimation
We experimentally find that the proposed iterative inference from conditional score estimation by conditional denoising autoencoders performs better than comparable models based on CRFs or those not using any explicit modeling of the conditional joint distribution of outputs.
Deep Complex Networks
Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models.
Ranked #3 on
Music Transcription
on MusicNet
Deep Learning for Patient-Specific Kidney Graft Survival Analysis
An accurate model of patient-specific kidney graft survival distributions can help to improve shared-decision making in the treatment and care of patients.
Learning to Compute Word Embeddings On the Fly
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare.
Ranked #48 on
Question Answering
on SQuAD1.1 dev
Gated Orthogonal Recurrent Units: On Learning to Forget
We present a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory.
Ranked #7 on
Question Answering
on bAbi
(Accuracy (trained on 1k) metric)
Plan, Attend, Generate: Character-level Neural Machine Translation with Planning in the Decoder
We investigate the integration of a planning mechanism into an encoder-decoder architecture with an explicit alignment for character-level machine translation.
Neural Models for Key Phrase Detection and Question Generation
We propose a two-stage neural model to tackle question generation from documents.
A Closer Look at Memorization in Deep Networks
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness.
Variance Regularizing Adversarial Learning
We hypothesize that this approach ensures a non-zero gradient to the generator, even in the limit of a perfect classifier.
Multiscale sequence modeling with a learned dictionary
We propose a generalization of neural network sequence models.
Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition
Layer normalization is a recently introduced technique for normalizing the activities of neurons in deep neural networks to improve the training speed and stability.
Plan, Attend, Generate: Character-Level Neural Machine Translation with Planning
We investigate the integration of a planning mechanism into an encoder-decoder architecture with attention.
Independently Controllable Factors
It has been postulated that a good representation is one that disentangles the underlying explanatory factors of variation.
Twin Networks: Matching the Future for Sequence Generation
We propose a simple technique for encouraging generative RNNs to plan ahead.
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem.
A Deep Reinforcement Learning Chatbot
By applying reinforcement learning to crowdsourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble.
The Consciousness Prior
To the extent that these assumptions are generally true (and the form of natural language seems consistent with them), they can form a useful prior for representation learning.
Improving speech recognition by revising gated recurrent units
First, we suggest to remove the reset gate in the GRU design, resulting in a more efficient single-gate architecture.
Residual Connections Encourage Iterative Inference
In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features.
Learning Independent Features with Adversarial Nets for Non-linear ICA
We propose to learn independent features with adversarial objectives which optimize such measures implicitly.
Generalization in Deep Learning
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature.
FigureQA: An Annotated Figure Dataset for Visual Reasoning
To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure.
Ranked #3 on
Visual Question Answering (VQA)
on FigureQA - test 1
Graph Attention Networks
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations.
Ranked #1 on
Node Classification
on Pubmed
(Validation metric)
Fraternal Dropout
We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout.
Ranked #28 on
Language Modelling
on Penn Treebank (Word Level)
Monaural Singing Voice Separation with Skip-Filtering Connections and Recurrent Inference of Time-Frequency Mask
Singing voice separation based on deep learning relies on the usage of time-frequency masking.
Sound Audio and Speech Processing
Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks
A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation.
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
The energy function is then modified so the model and data distributions match, with no guarantee on the number of steps required for the Markov chain to converge.
Three Factors Influencing Minima in SGD
In particular we find that the ratio of learning rate to batch size is a key determinant of SGD dynamics and of the width of the final minima, and that higher values of the ratio lead to wider minima and often better generalization.
ACtuAL: Actor-Critic Under Adversarial Learning
This makes it fundamentally difficult to train GANs with discrete data, as generation in this case typically involves a non-differentiable function.
Variational Bi-LSTMs
Bidirectional LSTMs (Bi-LSTMs) on the other hand model sequences along both forward and backward directions and are generally known to perform better at such tasks because they capture a richer representation of the data.
Z-Forcing: Training Stochastic Recurrent Networks
Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech.
Equivalence of Equilibrium Propagation and Recurrent Backpropagation
Recurrent Backpropagation and Equilibrium Propagation are supervised learning algorithms for fixed point recurrent neural networks which differ in their second phase.
Plan, Attend, Generate: Planning for Sequence-to-Sequence Models
We investigate the integration of a planning mechanism into sequence-to-sequence models using attention.
Measuring the tendency of CNNs to Learn Surface Statistical Regularities
The goal of this article is to measure the tendency of CNNs to learn surface statistical regularities of the dataset.
ObamaNet: Photo-realistic lip-sync from text
We present ObamaNet, the first architecture that generates both audio and synchronized photo-realistic lip-sync videos from any new text.
GibbsNet: Iterative Adversarial Inference for Deep Graphical Models
Directed latent variable models that formulate the joint distribution as $p(x, z) = p(z) p(x \mid z)$ have the advantage of fast and exact sampling.
Dendritic error backpropagation in deep cortical microcircuits
Animal behaviour depends on learning to associate sensory stimuli with the desired motor command.
Extending the Framework of Equilibrium Propagation to General Dynamics
The biological plausibility of the backpropagation algorithm has long been doubted by neuroscientists.
Boundary Seeking GANs
We introduce a method for training GANs with discrete data that uses the estimated difference measure from the discriminator to compute importance weights for generated samples, thus providing a policy gradient for training the generator.
Learning Generative Models with Locally Disentangled Latent Factors
One of the most successful techniques in generative models has been decomposing a complicated generation task into a series of simpler generation tasks.
Combining Model-based and Model-free RL via Multi-step Control Variates
Model-free deep reinforcement learning algorithms are able to successfully solve a wide range of continuous control tasks, but typically require many on-policy samples to achieve good performance.
