Search Results for author: Geoffrey Hinton

Found 31 papers, 19 papers with code

NASA Neural Articulated Shape Approximation

no code implementations ECCV 2020 Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi

Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics.

How to represent part-whole hierarchies in a neural network

4 code implementations25 Feb 2021 Geoffrey Hinton

Instead, it presents a single idea about representation which allows advances made by several different groups to be combined into an imaginary system called GLOM.

Representation Learning

Teaching with Commentaries

1 code implementation ICLR 2021 Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, Geoffrey Hinton

We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process.

Data Augmentation

Big Self-Supervised Models are Strong Semi-Supervised Learners

7 code implementations NeurIPS 2020 Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.

Fine-tuning Self-Supervised Image Classification +1

Neural Additive Models: Interpretable Machine Learning with Neural Nets

4 code implementations NeurIPS 2021 Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, Geoffrey Hinton

They perform similarly to existing state-of-the-art generalized additive models in accuracy, but are more flexible because they are based on neural nets instead of boosted trees.

Additive models Decision Making +1

Deflecting Adversarial Attacks

no code implementations18 Feb 2020 Yao Qin, Nicholas Frosst, Colin Raffel, Garrison Cottrell, Geoffrey Hinton

There has been an ongoing cycle where stronger defenses against adversarial attacks are subsequently broken by a more advanced defense-aware attack.

Subclass Distillation

no code implementations10 Feb 2020 Rafael Müller, Simon Kornblith, Geoffrey Hinton

By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data.

NASA: Neural Articulated Shape Approximation

no code implementations6 Dec 2019 Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi

Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics.

Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions

no code implementations ICLR 2020 Yao Qin, Nicholas Frosst, Sara Sabour, Colin Raffel, Garrison Cottrell, Geoffrey Hinton

Then, we diagnose the adversarial examples for CapsNets and find that the success of the reconstructive attack is highly related to the visual similarity between the source and target class.

When Does Label Smoothing Help?

2 code implementations NeurIPS 2019 Rafael Müller, Simon Kornblith, Geoffrey Hinton

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels.

Knowledge Distillation Speech Recognition +1

Cerberus: A Multi-headed Derenderer

no code implementations28 May 2019 Boyang Deng, Simon Kornblith, Geoffrey Hinton

To generalize to novel visual scenes with new viewpoints and new object poses, a visual system needs representations of the shapes of the parts of an object that are invariant to changes in viewpoint or pose.

Similarity of Neural Network Representations Revisited

7 code implementations ICML 2019 2019 Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton

We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation.

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss

2 code implementations5 Feb 2019 Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton

We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i. e., how close pairs of points from the same class are relative to pairs of points from different classes.

DARCCC: Detecting Adversaries by Reconstruction from Class Conditional Capsules

no code implementations16 Nov 2018 Nicholas Frosst, Sara Sabour, Geoffrey Hinton

In addition to being trained to classify images, the capsule model is trained to reconstruct the images from the pose parameters and identity of the correct top-level capsule.

Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search

no code implementations ACL 2018 Jamie Kiros, William Chan, Geoffrey Hinton

We introduce Picturebook, a large-scale lookup operation to ground language via {`}snapshots{'} of our physical world accessed through image search.

General Classification Image Retrieval +8

Distilling a Neural Network Into a Soft Decision Tree

6 code implementations27 Nov 2017 Nicholas Frosst, Geoffrey Hinton

They excel when the input data is high dimensional, the relationship between the input and the output is complicated, and the number of labeled training examples is large.

Classification General Classification

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

3 code implementations23 Jan 2017 Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean

In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters.

Language Modelling Machine Translation +1

Using Fast Weights to Attend to the Recent Past

4 code implementations NeurIPS 2016 Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs.

Distilling the Knowledge in a Neural Network

45 code implementations9 Mar 2015 Geoffrey Hinton, Oriol Vinyals, Jeff Dean

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.

Knowledge Distillation

Grammar as a Foreign Language

8 code implementations NeurIPS 2015 Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, Geoffrey Hinton

Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades.

Constituency Parsing

On the importance of initialization and momentum in deep learning

no code implementations Proceedings of the 30th International Conference on Machine Learning 2013 Ilya Sutskever, James Martens, George Dahl, Geoffrey Hinton

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum.

Deep Neural Networks for Acoustic Modeling in Speech Recognition

no code implementations Signal Processing Magazine 2012 Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, Brian Kingsbury

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input.

Speech Recognition

Cannot find the paper you are looking for? You can Submit a new open access paper.