Search Results for author:
Found 65 papers, 11 papers with code
Parameter Symmetry Breaking and Restoration Determines the Hierarchical Learning in AI Systems
The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems.
What if Eye...? Computationally Recreating Vision Evolution
First, we demonstrate computational evidence that task specific selection drives bifurcation in eye evolution - orientation tasks like navigation in a maze leads to distributed compound-type eyes while an object discrimination task leads to the emergence of high-acuity camera-type eyes.
Self-Assembly of a Biologically Plausible Learning Circuit
Over the last four decades, the amazing success of deep learning has been driven by the use of Stochastic Gradient Descent (SGD) as the main optimization technique.
On Generalization Bounds for Neural Networks with Low Rank Layers
While previous optimization results have suggested that deep neural networks tend to favour low-rank weight matrices, the implications of this inductive bias on generalization bounds remain underexplored.
Training the Untrainable: Introducing Inductive Bias via Representational Alignment
If the guide is trained, this transfers over part of the architectural prior and knowledge of the guide to the target.
Formation of Representations in Neural Networks
We then show that the breaking of CRH leads to the emergence of reciprocal power-law relations between R, W, and G, which we refer to as the Polynomial Alignment Hypothesis (PAH).
On the Power of Decision Trees in Auto-Regressive Language Modeling
This research reveals the unique computational abilities of ARDTs, aiming to broaden the architectural diversity in language model development.
How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD
In contrast, we demonstrate that while vanilla GD also approximates the target function, it requires an explicit regularization term to learn the support in the first layer.
System identification of neural systems: If we got it right, would we know?
The networks are compared to recordings of biological neurons, and good performance in reproducing neural responses is considered to support the model's validity.
Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs.
Iterative regularization in classification via hinge loss diagonal descent
In this paper, we focus on iterative regularization in the context of classification.
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks.
Neural-guided, Bidirectional Program Search for Abstraction and Reasoning
More specifically, we extend existing execution-guided program synthesis approaches with deductive reasoning based on function inverse semantics to enable a neural-guided bidirectional search algorithm.
Distribution of Classification Margins: Are All Data Equal?
Recent theoretical results show that gradient descent on deep neural networks under exponential loss functions locally maximizes classification margin, which is equivalent to minimizing the norm of the weight matrices under margin constraints.
The Effects of Image Distribution and Task on Adversarial Robustness
In this paper, we propose an adaptation to the area under the curve (AUC) metric to measure the adversarial robustness of a model over a particular $\epsilon$-interval $[\epsilon_0, \epsilon_1]$ (interval of adversarial perturbation strengths) that facilitates unbiased comparisons across models when they have different initial $\epsilon_0$ performance.
Explicit regularization and implicit bias in deep network classifiers trained with the square loss
Deep ReLU networks trained with the square loss have been observed to perform well in classification tasks.
CUDA-Optimized real-time rendering of a Foveated Visual System
The spatially-varying field of the human visual system has recently received a resurgence of interest with the development of virtual reality (VR) and neural networks.
Biologically Inspired Mechanisms for Adversarial Robustness
A convolutional neural network strongly robust to adversarial perturbations at reasonable computational and performance cost has not yet been demonstrated.
For interpolating kernel machines, minimizing the norm of the ERM solution minimizes stability
We study the average $\mbox{CV}_{loo}$ stability of kernel ridge-less regression and derive corresponding risk bounds.
Hierarchically Compositional Tasks and Deep Convolutional Networks
For object recognition we find, as expected, that scrambling does not affect the performance of shallow or deep fully connected networks contrary to the out-performance of convolutional networks.
Double descent in the condition number
In solving a system of $n$ linear equations in $d$ variables $Ax=b$, the condition number of the $n, d$ matrix $A$ measures how much errors in the data $b$ affect the solution $x$.
Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
In approximation theory both shallow and deep networks have been shown to approximate any continuous functions on a bounded domain at the expense of an exponential number of parameters (exponential in the dimensionality of the function).
Theory III: Dynamics and Generalization in Deep Networks
In particular, gradient descent induces a dynamics of the normalized weights which converge for $t \to \infty$ to an equilibrium which corresponds to a minimum norm (or maximum margin) solution.
Biologically-plausible learning algorithms can scale to large datasets
These results complement the study by Bartunov et al. (2018), and establish a new benchmark for future biologically plausible learning algorithms on more difficult datasets and more complex architectures.
A Surprising Linear Relationship Predicts Test Performance in Deep Networks
Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors?
Theory IIIb: Generalization in Deep Networks
Here we prove a similar result for nonlinear multilayer DNNs near zero minima of the empirical loss.
Approximate inference with Wasserstein gradient flows
We present a novel approximate inference method for diffusion processes, based on the Wasserstein gradient flow formulation of the diffusion.
An analysis of training and generalization errors in shallow and deep networks
We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure.
Theory of Deep Learning IIb: Optimization Properties of SGD
In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent.
Theory of Deep Learning III: explaining the non-overfitting puzzle
In this note, we show that the dynamics associated to gradient descent minimization of nonlinear networks is topologically equivalent, near the asymptotically stable minima of the empirical error, to linear gradient system in a quadratic potential with a degenerate (for square loss) or almost degenerate (for logistic or crossentropy loss) Hessian.
Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint.
Pruning Convolutional Neural Networks for Image Instance Retrieval
In this work, we focus on the problem of image instance retrieval with deep descriptors extracted from pruned Convolutional Neural Networks (CNN).
Do Deep Neural Networks Suffer from Crowding?
Also, for all tested networks, when trained on targets in isolation, we find that recognition accuracy of the networks decreases the closer the flankers are to the target and the more flankers there are.
Theory II: Landscape of the Empirical Risk in Deep Learning
Previous theoretical work on deep learning and neural network optimization tend to focus on avoiding saddle points and local minima.
Compression of Deep Neural Networks for Image Instance Retrieval
One major drawback of CNN-based {\it global descriptors} is that uncompressed deep neural network models require hundreds of megabytes of storage making them inconvenient to deploy in mobile applications or in custom hardware.
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review
The paper characterizes classes of functions for which deep learning can be exponentially better than shallow learning.
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning.
Deep vs. shallow networks : An approximation theory perspective
The paper announces new results for a non-smooth activation function - the ReLU function - used in present-day neural networks, as well as for the Gaussian networks.
View-tolerant face recognition and Hebbian learning imply mirror-symmetric neural tuning to head orientation
The primate brain contains a hierarchy of visual areas, dubbed the ventral stream, which rapidly computes object representations that are both specific for object identity and relatively robust against identity-preserving transformations like depth-rotations.
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex.
Nested Invariance Pooling and RBM Hashing for Image Instance Retrieval
The first one is Nested Invariance Pooling (NIP), a method inspired from i-theory, a mathematical theory for computing group invariant transformations with feed-forward neural networks.
Learning Functions: When Is Deep Better Than Shallow
While the universal approximation property holds both for hierarchical and shallow networks, we prove that deep (hierarchical) networks can approximate the class of compositional functions with the same accuracy as shallow networks but with exponentially lower number of training parameters as well as VC-dimension.
Group Invariant Deep Representations for Image Instance Retrieval
Based on a thorough empirical evaluation using several publicly available datasets, we show that our method is able to significantly and consistently improve retrieval results every time a new type of invariance is incorporated.
Foveation-based Mechanisms Alleviate Adversarial Examples
To see this, first, we report results in ImageNet that lead to a revision of the hypothesis that adversarial perturbations are a consequence of CNNs acting as a linear classifier: CNNs act locally linearly to changes in the image regions with objects recognized by the CNN, and in other regions the CNN may act non-linearly.
How Important is Weight Symmetry in Backpropagation?
Gradient backpropagation (BP) requires symmetric feedforward and feedback connections -- the same weights must be used for forward and backward passes.
Ranked #1 on
Handwritten Digit Recognition
on MNIST
(PERCENTAGE ERROR metric)
Holographic Embeddings of Knowledge Graphs
Learning embeddings of entities and relations is an efficient and versatile method to perform machine learning on relational data such as knowledge graphs.
Ranked #9 on
Link Prediction
on FB15k
Deep Convolutional Networks are Hierarchical Kernel Machines
In i-theory a typical layer of a hierarchical architecture consists of HW modules pooling the dot products of the inputs to the layer with the transformations of a few templates under a group.
Learning with a Wasserstein Loss
In this paper we develop a loss function for multi-label learning, based on the Wasserstein distance.
Learning with Group Invariant Features: A Kernel Perspective
Our analysis bridges invariant feature learning with kernel methods, as we show that this feature map defines an expected Haar integration kernel that is invariant to the specified group action.
Convex Learning of Multiple Tasks and their Structure
In this context a fundamental question is how to incorporate the tasks structure in the learning problem. We tackle this question by studying a general computational framework that allows to encode a-priori knowledge of the tasks structure in the form of a convex penalty; in this setting a variety of previously proposed methods can be recovered as special cases, including linear and non-linear approaches.
On Invariance and Selectivity in Representation Learning
We discuss data representation which can be learned automatically from data, are invariant to transformations, and at the same time selective, in the sense that two points have the same representation only if they are one the transformation of the other.
Unsupervised learning of clutter-resistant visual representations from natural videos
Populations of neurons in inferotemporal cortex (IT) maintain an explicit code for object identity that also tolerates transformations of object appearance e. g., position, scale, viewing angle [1, 2, 3].
Learning An Invariant Speech Representation
Recognition of speech, and in particular the ability to generalize and learn from small sets of labelled examples like humans do, depends on an appropriate representation of the acoustic input.
Neural tuning size is a key factor underlying holistic face processing
The main aim of this work is to further the fundamental understanding of what causes the visual processing of faces to be different from that of objects.
Computational role of eccentricity dependent cortical magnification
From the slope of the inverse of the magnification factor, M-theory predicts a cortical "fovea" in V1 in the order of $40$ by $40$ basic units at each receptive field size -- corresponding to a foveola of size around $26$ minutes of arc at the highest resolution, $\approx 6$ degrees at the lowest resolution.
A Deep Representation for Invariance And Music Classification
We present the main theoretical and computational aspects of a framework for unsupervised learning of invariant audio representations, empirically evaluated on music genre classification.
Learning invariant representations and applications to face verification
Next, we apply the model to non-affine transformations: as expected, it performs well on face verification tasks requiring invariance to the relatively smooth transformations of 3D rotation-in-depth and changes in illumination direction.
Neural representation of action sequences: how far can a simple snippet-matching model take us?
The macaque Superior Temporal Sulcus (STS) is a brain area that receives and integrates inputs from both the ventral and dorsal visual processing streams (thought to specialize in form and motion processing respectively).
Unsupervised Learning of Invariant Representations in Hierarchical Architectures
It also suggests that the main computational goal of the ventral stream of visual cortex is to provide a hierarchical representation of new objects/images which is invariant to transformations, stable, and discriminative for recognition---and that this representation may be continuously learned in an unsupervised way during development and visual experience.
Can a biologically-plausible hierarchy effectively replace face detection, alignment, and recognition pipelines?
The standard approach to unconstrained face recognition in natural photographs is via a detection, alignment, recognition pipeline.
On Learnability, Complexity and Stability
We consider the fundamental question of learnability of a hypotheses class in the supervised learning setting and in the general learning setting introduced by Vladimir Vapnik.
Multiclass Learning with Simplex Coding
In this paper we dicuss a novel framework for multiclass learning, defined by a suitable coding/decoding strategy, namely the simplex coding, that allows to generalize to multiple classes a relaxation approach commonly used in binary classification.
Learning Manifolds with K-Means and K-Flats
We study the problem of estimating a manifold from random samples.
Why The Brain Separates Face Recognition From Object Recognition
Many studies have uncovered evidence that visual cortex contains specialized regions involved in processing faces but not other object classes.
On Invariance in Hierarchical Models
A goal of central importance in the study of hierarchical models for object recognition -- and indeed the visual cortex -- is that of understanding quantitatively the trade-off between invariance and selectivity, and how invariance and discrimination properties contribute towards providing an improved representation useful for learning from data.
