Search Results for author:
Found 18 papers, 2 papers with code
Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
Diverse studies in systems neuroscience begin with extended periods of training known as 'shaping' procedures.
The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
Reinforcement learning (RL) algorithms have proven transformative in a range of domains.
Optimal transfer protocol by incremental layer defrosting
The simplest transfer learning protocol is based on ``freezing" the feature-extractor layers of a network pre-trained on a data-rich source task, and then adapting only the last layers to a data-poor target task.
Bias-inducing geometries: an exactly solvable data model with fairness implications
Then, we consider a novel mitigation strategy based on a matched inference approach, consisting in the introduction of coupled learning models.
Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
Continual learning - learning new tasks in sequence while maintaining performance on old tasks - remains particularly challenging for artificial neural networks.
An Analytical Theory of Curriculum Learning in Teacher-Student Networks
To study the former, we provide an exact description of the online learning setting, confirming the long-standing experimental observation that curricula can modestly speed up learning.
Probing transfer learning with a model of synthetic correlated datasets
Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task.
Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
The optimization step in many machine learning problems rarely relies on vanilla gradient descent but it is common practice to use momentum-based accelerated methods.
Epidemic mitigation by statistical inference from contact tracing data
We conclude that probabilistic risk estimation is capable to enhance performance of digital contact tracing and should be considered in the currently developed mobile applications.
Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$.
Post-Workshop Report on Science meets Engineering in Deep Learning, NeurIPS 2019, Vancouver
Science meets Engineering in Deep Learning took place in Vancouver as part of the Workshop section of NeurIPS 2019.
Winning the competition: enhancing counter-contagion in SIS-like epidemic processes
In this paper we consider the epidemic competition between two generic diffusion processes, where each competing side is represented by a different state of a stochastic process.
Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval
Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem.
Thresholds of descending algorithms in inference problems
We review recent works on analyzing the dynamics of gradient-based algorithms in a prototypical statistical inference problem.
Who is Afraid of Big Bad Minima? Analysis of gradient-flow in spiked matrix-tensor models
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a quantitative theory explaining this behaviour in a spiked matrix-tensor model. Our framework is based on the Kac-Rice analysis of stationary points and a closed-form analysis of gradient-flow originating from statistical physics.
Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Model
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones.
Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models
In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor model.
Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference
Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference.
