Search Results for author:
Found 74 papers, 43 papers with code
How Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad
Can Transformers predict new syllogisms by composing established ones?
What Algorithms can Transformers Learn? A Study in Length Generalization
Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity.
When can transformers reason with abstract symbols?
We investigate the capabilities of transformer models on relational reasoning tasks.
Adaptivity and Modularity for Efficient Generalization Over Task Complexity
We investigate how the use of a mechanism for adaptive and modular computation in transformers facilitates the learning of tasks that demand generalization over the number of sequential computation steps (i. e., the depth of the computation graph).
Boolformer: Symbolic Regression of Logic Functions with Transformers
In this work, we introduce Boolformer, the first Transformer architecture trained to perform end-to-end symbolic regression of Boolean functions.
Transformers learn through gradual rank increase
Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.
Generalization on the Unseen, Logic Reasoning and Degree Curriculum
This paper considers the learning of logical (Boolean) functions with focus on the generalization on the unseen (GOTU) setting, a strong case of out-of-distribution generalization.
Continuous Soft Pseudo-Labeling in ASR
Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently emerged as a powerful strategy for semi-supervised learning in speech recognition.
Continuous Pseudo-Labeling from the Start
Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures
More generally, the paper considers the learning of logical functions with gradient descent (GD) on neural networks.
Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
In PVR, this is done by having one part of the task input act as a pointer, giving instructions on a different input location, which forms the output.
Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
Attentional mechanisms are order-invariant.
Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss
Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics.
NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
Understanding generalization in deep learning is arguably one of the most important questions in deep learning.
Data Augmentation via Structured Adversarial Perturbations
The advantage of adversarial augmentation is that it replaces sampling with the use of a single, calculated perturbation that maximally increases the loss.
Characterising Bias in Compressed Models
However, overall accuracy hides disproportionately high errors on a small subset of examples; we call this subset Compression Identified Exemplars (CIE).
Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders
While we focus on interface layout prediction, our model can be generally applicable for other layout prediction problems that involve tree structures and 2-dimensional placements.
Fantastic Generalization Measures and Where to Find Them
We present the first large scale study of generalization in deep networks.
Self-Imitation Learning via Trajectory-Conditioned Policy for Hard-Exploration Tasks
We propose a new method of learning a trajectory-conditioned policy to imitate diverse trajectories from the agent's own past experiences and show that such self-imitation helps avoid myopic behavior and increases the chance of finding a globally optimal solution for hard-exploration tasks, especially when there are misleading rewards.
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.
Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
Reinforcement learning with sparse rewards is challenging because an agent can rarely obtain non-zero rewards and hence, gradient-based optimization of parameterized policies can be incremental and slow.
Parallel Scheduled Sampling
Experimentally, we find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forced training.
A Closed-Form Learned Pooling for Deep Classification Networks
This operator can learn a strict super-set of what can be learned by average pooling or convolutions.
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
Furthermore, Cluster-GCN allows us to train much deeper GCN without much time and memory overhead, which leads to improved prediction accuracy---using a 5-layer Cluster-GCN, we achieve state-of-the-art test F1 score 99. 36 on the PPI dataset, while the previous best result was 98. 71 by [16].
Ranked #1 on
Node Classification
on Amazon2M
Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure
The Gestalt laws of perceptual organization, which describe how visual elements in an image are grouped and interpreted, have traditionally been thought of as innate despite their ecological validity.
Transfusion: Understanding Transfer Learning for Medical Imaging
Investigating the learned representations and features, we find that some of the differences from transfer learning are due to the over-parametrization of standard models rather than sophisticated feature reuse.
Identity Crisis: Memorization and Generalization under Extreme Overparameterization
We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task.
Are All Layers Created Equal?
Morally, layers of large deep neural networks can be categorized as either "robust" or "critical".
Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need
Large datasets have been crucial to the success of deep learning models in the recent years, which keep performing better as they are trained with more labelled data.
Unsupervised speech representation learning using WaveNet autoencoders
We consider the task of unsupervised extraction of meaningful latent representations of speech by applying autoencoding neural networks to speech waveforms.
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs
The concept of conditional computation for deep nets has been proposed previously to improve model performance by selectively using only parts of the model conditioned on the sample it is processing.
Area Attention
We propose area attention: a way to attend to areas in the memory, where each area contains a group of items that are structurally adjacent, e. g., spatially for a 2D memory such as images, or temporally for a 1D memory such as natural language sentences.
Predicting the Generalization Gap in Deep Networks with Margin Distributions
In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap.
Insights on representational similarity in neural networks with canonical correlation
Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, even across groups of networks trained on identical tasks, and over the course of training.
A Study on Overfitting in Deep Reinforcement Learning
We conclude with a general discussion on overfitting in RL and a study of the generalization behaviors from the perspective of inductive bias.
Adversarial Attacks and Defences Competition
To accelerate research on adversarial examples and robustness of machine learning classifiers, Google Brain organized a NIPS 2017 competition that encouraged researchers to develop new methods to generate adversarial examples as well as to develop new ways to defend against them.
Tensor2Tensor for Neural Machine Translation
Tensor2Tensor is a library for deep learning models that is well-suited for neural machine translation and includes the reference implementation of the state-of-the-art Transformer model.
Large Margin Deep Networks for Classification
We present a formulation of deep learning that aims at producing a large margin classifier.
Fast Decoding in Sequence Models using Discrete Latent Variables
Finally, we evaluate our model end-to-end on the task of neural machine translation, where it is an order of magnitude faster at decoding than comparable autoregressive models.
Discrete Autoencoders for Sequence Models
We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space.
On Using Backpropagation for Speech Texture Generation and Voice Conversion
Inspired by recent work on neural network image generation which rely on backpropagation towards the network inputs, we present a proof-of-concept system for speech texture synthesis and voice conversion based on two mechanisms: approximate inversion of the representation learned by a speech recognition neural network, and on matching statistics of neuron activations between different source and target utterances.
Time-Dependent Representation for Neural Event Sequence Prediction
Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization.
Device Placement Optimization with Reinforcement Learning
Key to our method is the use of a sequence-to-sequence model to predict which subsets of operations in a TensorFlow graph should run on which of the available devices.
N-gram Language Modeling using Recurrent Neural Network Estimation
Experiments on a small corpus (UPenn Treebank, one million words of training data and 10k vocabulary) have found the LSTM cell with dropout to be the best model for encoding the $n$-gram state when compared with feed-forward and vanilla RNN models.
Tacotron: Towards End-to-End Speech Synthesis
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.
Ranked #5 on
Speech Synthesis
on North American English
Sharp Minima Can Generalize For Deep Nets
Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice.
Learning to Remember Rare Events
We present a large-scale life-long memory module for use in deep learning.
Revisiting Distributed Synchronous SGD
Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony.
Context-aware Captions from Context-agnostic Supervision
We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation).
An Online Sequence-to-Sequence Model Using Partial Conditioning
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Neural Combinatorial Optimization with Reinforcement Learning
Despite the computational expense, without much engineering and heuristic designing, Neural Combinatorial Optimization achieves close to optimal results on 2D Euclidean graphs with up to 100 nodes.
Understanding deep learning requires rethinking generalization
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Adversarial Machine Learning at Scale
Adversarial examples are malicious inputs designed to fool machine learning models.
Can Active Memory Replace Attention?
Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years.
Ranked #54 on
Machine Translation
on WMT2014 English-French
Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing.
Reward Augmented Maximum Likelihood for Neural Structured Prediction
A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation.
Adversarial examples in the physical world
Up to now, all previous work have assumed a threat model in which the adversary can feed data directly into the machine learning classifier.
Density estimation using Real NVP
Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning.
Ranked #24 on
Image Generation
on ImageNet 32x32
(bpd metric)
Revisiting Distributed Synchronous SGD
Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony.
Generating Sentences from a Continuous Space
The standard recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global sentence representation.
Order Matters: Sequence to sequence for sets
Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks.
A Neural Transducer
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
End-to-End Text-Dependent Speaker Verification
In this paper we present a data-driven, integrated approach to speaker verification, which maps a test utterance and a few reference utterances directly to a single score for verification and jointly optimizes the system's components using the same evaluation protocol and metric as at test time.
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Recurrent Neural Networks can be trained to produce sequences of tokens given some input, as exemplified by recent results in machine translation and image captioning.
Learning Semantic Relationships for Better Action Retrieval in Images
Human actions capture a wide variety of interactions between people and objects.
Show and Tell: A Neural Image Caption Generator
Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions.
Ranked #3 on
Image Retrieval with Multi-Modal Query
on MIT-States
Zero-Shot Learning by Convex Combination of Semantic Embeddings
In other cases the semantic embedding space is established by an independent natural language processing task, and then the image transformation into that space is learned in a second stage.
Ranked #8 on
Multi-label zero-shot learning
on Open Images V4
Using Web Co-occurrence Statistics for Improving Image Categorization
Albeit the simplicity of the resulting optimization problem, it is effective in improving both recognition and localization accuracy.
DeViSE: A Deep Visual-Semantic Embedding Model
Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories.
Ranked #14 on
Zero-Shot Action Recognition
on Kinetics
Large-Scale Music Annotation and Retrieval: Learning to Rank in Joint Semantic Spaces
Music prediction tasks range from predicting tags given a song or clip of audio, predicting the name of the artist, or predicting related songs given a song, clip, artist name or tag.
Label Embedding Trees for Large Multi-Class Tasks
Multi-class classification becomes challenging at test time when the number of classes is very large and testing against every possible class can become computationally infeasible.
An Online Algorithm for Large Scale Image Similarity Learning
We describe OASIS, a method for learning pairwise similarity that is fast and scales linearly with the number of objects and the number of non-zero features.
Group Sparse Coding
Bag-of-words document representations are often used in text, image and video processing.
