Search Results for author:
Found 21 papers, 3 papers with code
Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
We also show that the dense regions are surprisingly accessible by simple learning protocols, and that these synaptic configurations are robust to perturbations and generalize better than typical solutions.
Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
We introduce a novel Entropy-driven Monte Carlo (EdMC) strategy to efficiently sample solutions of random Constraint Satisfaction Problems (CSPs).
Learning may need only a few bits of synaptic precision
Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states.
Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
We define a novel measure, which we call the "robust ensemble" (RE), which suppresses trapping by isolated configurations and amplifies the role of these dense regions.
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape.
Efficiency of quantum versus classical annealing in non-convex learning problems
Their energy landscapes is dominated by local minima that cause exponential slow down of classical thermal annealers while simulated quantum annealing converges efficiently to rare dense regions of optimal solutions.
Parle: parallelizing stochastic gradient descent
We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters.
On the role of synaptic stochasticity in training low-precision neural networks
Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes.
Shaping the learning landscape in neural networks around wide flat minima
In the case of SGD and cross-entropy loss, we show that a slow reduction of the norm of the weights along the learning process also leads to WFM.
Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems.
Natural representation of composite data with replicated autoencoders
Generative processes in biology and other fields often produce data that can be regarded as resulting from a composition of basic features.
Clustering of solutions in the symmetric binary perceptron
The geometrical features of the (non-convex) loss landscape of neural network models are crucial in ensuring successful optimization and, most importantly, the capability to generalize well.
Entropic gradient descent algorithms and wide flat minima
The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time.
Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
We analyze the connection between minimizers with good generalizing properties and high local entropy regions of a threshold-linear classifier in Gaussian mixtures with the mean squared error loss function.
Unveiling the structure of wide flat minima in neural networks
The success of deep learning has revealed the application potential of neural networks across the sciences and opened up fundamental theoretical problems.
Learning through atypical "phase transitions" in overparameterized neural networks
Current deep neural networks are highly overparameterized (up to billions of connection weights) and nonlinear.
Deep learning via message passing algorithms based on belief propagation
Message-passing algorithms based on the Belief Propagation (BP) equations constitute a well-known distributed computational scheme.
Native state of natural proteins optimises local entropy
The differing ability of polypeptide conformations to act as the native state of proteins has long been rationalized in terms of differing kinetic accessibility or thermodynamic stability.
Quantum Approximate Optimization Algorithm applied to the binary perceptron
We apply digitized Quantum Annealing (QA) and Quantum Approximate Optimization Algorithm (QAOA) to a paradigmatic task of supervised learning in artificial neural networks: the optimization of synaptic weights for the binary perceptron.
Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry
This lets us derive a meaningful notion of the flatness of minimizers and of the geodesic paths connecting them.
Typical and atypical solutions in non-convex neural networks with discrete and continuous weights
We analyze the geometry of the landscape of solutions in both models and find important similarities and differences.
