Search Results for author:
Found 19 papers, 13 papers with code
Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime
We demonstrate that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization.
ODEFormer: Symbolic Regression of Dynamical Systems with Transformers
We introduce ODEFormer, the first transformer able to infer multidimensional ordinary differential equation (ODE) systems in symbolic form from the observation of a single solution trajectory.
Boolformer: Symbolic Regression of Logic Functions with Transformers
In this work, we introduce Boolformer, the first Transformer architecture trained to perform end-to-end symbolic regression of Boolean functions.
Length Generalization in Arithmetic Transformers
We find that relative position embeddings enable length generalization for simple tasks, such as addition: models trained on $5$-digit numbers can perform $15$-digit sums.
End-to-end symbolic regression with transformers
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function.
Optimal learning rate schedules in high-dimensional non-convex optimization problems
In this case, it is optimal to keep a large learning rate during the exploration phase to escape the non-convex region as quickly as possible, then use the convex criterion $\beta=1$ to converge rapidly to the solution.
Deep Symbolic Regression for Recurrent Sequences
Symbolic regression, i. e. predicting a function from the observation of its values, is well-known to be a challenging task.
Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Finally, we experiment initializing the T-CNN from a partially trained CNN, and find that it reaches better performance than the corresponding hybrid model trained from scratch, while reducing training time.
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information.
Ranked #523 on
Image Classification
on ImageNet
On the interplay between data structure and loss function in classification problems
One of the central puzzles in modern machine learning is the ability of heavily overparametrized models to generalize well.
Align, then memorise: the dynamics of learning with feedback alignment
Direct Feedback Alignment (DFA) is emerging as an efficient and biologically plausible alternative to the ubiquitous backpropagation algorithm for training deep neural networks.
Conditioned Text Generation with Transfer for Closed-Domain Dialogue Systems
Scarcity of training data for task-oriented dialogue systems is a well known problem that is usually tackled with costly and time-consuming manual data annotation.
Triple descent and the two kinds of overfitting: Where & why do they appear?
We show that this peak is implicitly regularized by the nonlinearity, which is why it only becomes salient at high noise and is weakly affected by explicit regularization.
Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant.
Conditioned Query Generation for Task-Oriented Dialogue Systems
Scarcity of training data for task-oriented dialogue systems is a well known problem that is usually tackled with costly and time-consuming manual data annotation.
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
The aim of this work is to understand this fact through the lens of dynamics in the loss landscape.
Scaling description of generalization with number of parameters in deep learning
At this threshold, we argue that $\|f_{N}\|$ diverges.
A jamming transition from under- to over-parametrization affects loss landscape and generalization
We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved.
The jamming transition as a paradigm to understand the loss landscape of deep neural networks
In the vicinity of this transition, properties of the curvature of the minima of the loss are critical.
