Search Results for author: Geoffrey Hinton

Found 42 papers, 28 papers with code

NASA Neural Articulated Shape Approximation

no code implementations ECCV 2020 Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi

Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics.

The Forward-Forward Algorithm: Some Preliminary Investigations

16 code implementations NA 2022 Geoffrey Hinton

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation.

Meta-Learning Fast Weight Language Models

no code implementations5 Dec 2022 Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, Mohammad Norouzi

Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance.

Language Modelling Meta-Learning

Gaussian-Bernoulli RBMs Without Tears

1 code implementation19 Oct 2022 Renjie Liao, Simon Kornblith, Mengye Ren, David J. Fleet, Geoffrey Hinton

We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations.

Scaling Forward Gradient With Local Losses

1 code implementation7 Oct 2022 Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton

Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks.

Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

6 code implementations8 Aug 2022 Ting Chen, Ruixiang Zhang, Geoffrey Hinton

The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits.

Image Captioning Image Generation

A Unified Sequence Interface for Vision Tasks

1 code implementation15 Jun 2022 Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, Geoffrey Hinton

Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization.

Image Captioning Instance Segmentation +2

How to represent part-whole hierarchies in a neural network

5 code implementations25 Feb 2021 Geoffrey Hinton

Instead, it presents a single idea about representation which allows advances made by several different groups to be combined into an imaginary system called GLOM.

Representation Learning

Teaching with Commentaries

1 code implementation ICLR 2021 Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, Geoffrey Hinton

We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process.

Data Augmentation

Big Self-Supervised Models are Strong Semi-Supervised Learners

8 code implementations NeurIPS 2020 Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.

Self-Supervised Image Classification Semi-Supervised Image Classification

Neural Additive Models: Interpretable Machine Learning with Neural Nets

6 code implementations NeurIPS 2021 Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, Geoffrey Hinton

They perform similarly to existing state-of-the-art generalized additive models in accuracy, but are more flexible because they are based on neural nets instead of boosted trees.

Additive models BIG-bench Machine Learning +3

Deflecting Adversarial Attacks

no code implementations18 Feb 2020 Yao Qin, Nicholas Frosst, Colin Raffel, Garrison Cottrell, Geoffrey Hinton

There has been an ongoing cycle where stronger defenses against adversarial attacks are subsequently broken by a more advanced defense-aware attack.

Subclass Distillation

no code implementations10 Feb 2020 Rafael Müller, Simon Kornblith, Geoffrey Hinton

By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data.

NASA: Neural Articulated Shape Approximation

no code implementations6 Dec 2019 Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi

Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics.

Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions

no code implementations ICLR 2020 Yao Qin, Nicholas Frosst, Sara Sabour, Colin Raffel, Garrison Cottrell, Geoffrey Hinton

Then, we diagnose the adversarial examples for CapsNets and find that the success of the reconstructive attack is highly related to the visual similarity between the source and target class.

When Does Label Smoothing Help?

3 code implementations NeurIPS 2019 Rafael Müller, Simon Kornblith, Geoffrey Hinton

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels.

Image Classification Knowledge Distillation +3

Cerberus: A Multi-headed Derenderer

no code implementations28 May 2019 Boyang Deng, Simon Kornblith, Geoffrey Hinton

To generalize to novel visual scenes with new viewpoints and new object poses, a visual system needs representations of the shapes of the parts of an object that are invariant to changes in viewpoint or pose.

Similarity of Neural Network Representations Revisited

9 code implementations ICML 2019 2019 Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton

We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation.

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss

4 code implementations5 Feb 2019 Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton

We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i. e., how close pairs of points from the same class are relative to pairs of points from different classes.

Classification

DARCCC: Detecting Adversaries by Reconstruction from Class Conditional Capsules

no code implementations16 Nov 2018 Nicholas Frosst, Sara Sabour, Geoffrey Hinton

In addition to being trained to classify images, the capsule model is trained to reconstruct the images from the pose parameters and identity of the correct top-level capsule.

Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search

no code implementations ACL 2018 Jamie Kiros, William Chan, Geoffrey Hinton

We introduce Picturebook, a large-scale lookup operation to ground language via {`}snapshots{'} of our physical world accessed through image search.

General Classification Image Retrieval +8

Distilling a Neural Network Into a Soft Decision Tree

6 code implementations27 Nov 2017 Nicholas Frosst, Geoffrey Hinton

They excel when the input data is high dimensional, the relationship between the input and the output is complicated, and the number of labeled training examples is large.

General Classification

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

4 code implementations23 Jan 2017 Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean

In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters.

Computational Efficiency Language Modelling +2

Using Fast Weights to Attend to the Recent Past

3 code implementations NeurIPS 2016 Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs.

Distilling the Knowledge in a Neural Network

60 code implementations9 Mar 2015 Geoffrey Hinton, Oriol Vinyals, Jeff Dean

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.

Knowledge Distillation

Grammar as a Foreign Language

8 code implementations NeurIPS 2015 Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, Geoffrey Hinton

Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades.

Constituency Parsing

On the importance of initialization and momentum in deep learning

no code implementations Proceedings of the 30th International Conference on Machine Learning 2013 Ilya Sutskever, James Martens, George Dahl, Geoffrey Hinton

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum.

Second-order methods

Deep Neural Networks for Acoustic Modeling in Speech Recognition

no code implementations Signal Processing Magazine 2012 Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, Brian Kingsbury

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input.

speech-recognition Speech Recognition

Visualizing Data using t-SNE

1 code implementation JMLR 2008 Laurens van der Maaten, Geoffrey Hinton

The technique is a variation of Stochastic Neighbor Embedding (Hinton and Roweis, 2002) that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map.

Dimensionality Reduction

Cannot find the paper you are looking for? You can Submit a new open access paper.