Search Results for author:
Found 10 papers, 1 papers with code
Falcon Mamba: The First Competitive Attention-free 7B Language Model
It is on par with Gemma 7B and outperforms models with different architecture designs, such as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B.
SGD with memory: fundamental properties and stochastic acceleration
In the non-stochastic setting, the optimal exponent $\xi$ in the loss convergence $L_t\sim C_Lt^{-\xi}$ is double that in plain GD and is achievable using Heavy Ball (HB) with a suitable schedule; this no longer works in the presence of mini-batch noise.
Falcon2-11B Technical Report
We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model.
Generalization error of spectral algorithms
In the present work, we consider the training of kernels with a family of $\textit{spectral algorithms}$ specified by profile $h(\lambda)$, and including KRR and GD as special cases.
Efficient Conformal Prediction under Data Heterogeneity
Conformal Prediction (CP) stands out as a robust framework for uncertainty quantification, which is crucial for ensuring the reliability of predictions.
A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta
Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models.
Embedded Ensembles: Infinite Width Limit and Operating Regimes
In this limit, we identify two ensemble regimes - independent and collective - depending on the architecture and initialization strategy of ensemble models.
Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions
In this paper, we propose a new spectral condition providing tighter upper bounds for problems with power law optimization trajectories.
Explicit loss asymptotics in the gradient descent training of neural networks
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss values.
Universal scaling laws in the gradient descent training of neural networks
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss values.
