Capsule networks aim to parse images into a hierarchy of objects, parts and relations.
In the second stage, SCAE predicts parameters of a few object capsules, which are then used to reconstruct part poses.
Ranked #2 on Unsupervised MNIST on MNIST
Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights.
Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance.
Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made.
A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships.
Ranked #3 on Image Classification on smallNORB
We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters.
Ranked #1 on Image Classification on MultiMNIST
We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012).
We present a framework for efficient inference in structured image models that explicitly reason about objects.
Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients.
Ranked #23 on Sequential Image Classification on Sequential MNIST
We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents.
We trained a large, deep convolutional neural network to classify the 1. 3 million high-resolution images in the LSVRC-2010 ImageNet training set into the 1000 different classes.
Ranked #4 on Graph Classification on HIV-fMRI-77
We describe how the pre-training algorithm for Deep Boltzmann Machines (DBMs) is related to the pre-training algorithm for Deep Belief Networks and we show that under certain conditions, the pre-training procedure improves the variational lower bound of a two-hidden-layer DBM.
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data.
Ranked #178 on Image Classification on CIFAR-10
Straightforward application of Deep Belief Nets (DBNs) to acoustic modeling produces a rich distributed representation of speech data that is useful for recognition and yields impressive results on the speaker-independent TIMIT phone recognition task.
We describe a log-bilinear" model that computes class probabilities by combining an input vector multiplicatively with a vector of binary latent variables.
Probabilistic models of natural images are usually evaluated by measuring performance on rather indirect tasks, such as denoising and inpainting.
We describe a model based on a Boltzmann machine with third-order connections that can learn how to accumulate information about a shape over several fixations.
To achieve this, we define the notion of a semantic output code classifier (SOC) which utilizes a knowledge base of semantic properties of $Y$ to extrapolate to novel classes.
Each member of the family models the probability distribution of documents of a specific length as a product of topic-specific distributions rather than as a mixture and this gives much better generalization than Latent Dirichlet Allocation for modeling the log probabilities of held-out documents.
Our model achieves 6. 5% error on the test set, which is close to the best published result for NORB (5. 9%) using a convolutional neural net that has built-in knowledge of translation invariance.
The Temporal Restricted Boltzmann Machine (TRBM) is a probabilistic model for sequences that is able to successfully model (i. e., generate nice-looking samples of) several very high dimensional sequences, such as motion capture data and the pixels of low resolution videos of balls bouncing in a box.
We describe an efficient learning procedure for multilayer generative models that combine the best aspects of Markov random fields and deep, directed belief nets.
We show how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process.
High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors.
Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
The rule, called the generalized delta rule, is a simple scheme for implementing a gradient descent method for finding weights that minimize the sum squared error of the system's performance.