Search Results for author:
Found 46 papers, 7 papers with code
Phi-4 Technical Report
We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality.
Small Language Models for Application Interactions: A Case Study
We study the efficacy of Small Language Models (SLMs) in facilitating application usage through natural language interactions.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
Positional Description Matters for Transformers Arithmetic
For (i) we train a small model on a small dataset (100M parameters and 300k samples) with remarkable aptitude in (direct, no scratchpad) 15 digits multiplication and essentially perfect up to 12 digits, while usual training in this context would give a model failing at 4 digits multiplication.
Textbooks Are All You Need II: phi-1.5 technical report
We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce coherent English -- and the follow-up work on \textbf{phi-1}, a 1. 3 billion parameter model with Python coding performance close to the state-of-the-art.
Ranked #16 on
Question Answering
on SIQA
Textbooks Are All You Need
Despite this small scale, phi-1 attains pass@1 accuracy 50. 6% on HumanEval and 55. 5% on MBPP.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models.
Ranked #34 on
Arithmetic Reasoning
on GSM8K
Learning threshold neurons via the "edge of stability"
For these models, we provably establish the edge of stability phenomenon and discover a sharp phase transition for the step size below which the neural network fails to learn "threshold-like" neurons (i. e., neurons with a non-zero first-layer bias).
Unveiling Transformers with LEGO: a synthetic reasoning task
We study how the trained models eventually succeed at the task, and in particular, we manage to understand some of the attention heads as well as how the information flows in the network.
Data Augmentation as Feature Manipulation
In this work we consider another angle, and we study the effect of data augmentation on the dynamic of the learning process.
Adversarial Examples in Multi-Layer Random ReLU Networks
We consider the phenomenon of adversarial examples in ReLU networks with independent gaussian parameters.
A Universal Law of Robustness via Isoperimetry
Classically, data interpolation with a parametrized model class is possible as long as the number of parameters is larger than the number of equations to be satisfied.
A single gradient step finds adversarial examples on random two-layers neural networks
Daniely and Schacham recently showed that gradient descent finds adversarial examples on random undercomplete two-layers ReLU neural networks.
Cooperative and Stochastic Multi-Player Multi-Armed Bandit: Optimal Regret With Neither Communication Nor Collisions
We consider the cooperative multi-player version of the stochastic multi-armed bandit problem.
A law of robustness for two-layers neural networks
We make a precise conjecture that, for any Lipschitz activation function and for most datasets, any two-layers neural network with $k$ neurons that perfectly fit the data must have its Lipschitz constant larger (up to a constant) than $\sqrt{n/k}$ where $n$ is the number of datapoints.
Network size and weights size for memorization with two-layers neural networks
In contrast we propose a new training procedure for ReLU networks, based on complex (as opposed to real) recombination of the neurons, for which we show approximate memorization with both $O\left(\frac{n}{d} \cdot \frac{\log(1/\epsilon)}{\epsilon}\right)$ neurons, as well as nearly-optimal size of the weights.
Online Multiserver Convex Chasing and Optimization
We introduce the problem of $k$-chasing of convex functions, a simultaneous generalization of both the famous k-server problem in $R^d$, and of the problem of chasing convex bodies and functions.
Online Learning for Active Cache Synchronization
Existing multi-armed bandit (MAB) models make two implicit assumptions: an arm generates a payoff only when it is played, and the agent observes every payoff that is generated.
Coordination without communication: optimal regret in two players multi-armed bandits
We consider two agents playing simultaneously the same stochastic three-armed bandit problem.
How to trap a gradient flow
This viewpoint was explored in 1993 by Vavasis, who proposed an algorithm which, for any fixed finite dimension $d$, improves upon the $O(1/\varepsilon^2)$ oracle complexity of gradient descent.
Complexity of Highly Parallel Non-Smooth Convex Optimization
Namely we consider optimization algorithms interacting with a highly parallel gradient oracle, that is one that can answer $\mathrm{poly}(d)$ gradient queries in parallel.
Non-Stochastic Multi-Player Multi-Armed Bandits: Optimal Rate With Collision Information, Sublinear Without
We consider the non-stochastic version of the (cooperative) multi-player multi-armed bandit problem.
First-Order Bayesian Regret Analysis of Thompson Sampling
Second we replace the entropy over combinatorial actions by a coordinate entropy, which allows us to obtain the first optimal worst-case bound for Thompson Sampling in the combinatorial setting.
Improved Path-length Regret Bounds for Bandits
We study adaptive regret bounds in terms of the variation of the losses (the so-called path-length bounds) for both multi-armed bandit and more generally linear bandit.
Adversarial Examples from Cryptographic Pseudo-Random Generators
In our recent work (Bubeck, Price, Razenshteyn, arXiv:1805. 10204) we argued that adversarial examples in machine learning might be due to an inherent computational hardness of the problem.
Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
Under the global regularity assumption, we provide a simple yet efficient algorithm called distributed randomized smoothing (DRS) based on a local smoothing of the objective function, and show that DRS is within a $d^{1/4}$ multiplicative factor of the optimal convergence rate, where $d$ is the underlying dimension.
Optimization and Control
Adversarial examples from computational constraints
First we prove that, for a broad set of classification tasks, the mere existence of a robust classifier implies that it can be found by a possibly exponential-time algorithm with relatively few training examples.
Make the Minority Great Again: First-Order Regret Bound for Contextual Bandits
Regret bounds in online learning compare the player's performance to $L^*$, the optimal performance in hindsight with a fixed strategy.
Sparsity, variance and curvature in multi-armed bandits
In (online) learning theory the concepts of sparsity, variance and curvature are well-understood and are routinely used to obtain refined regret and generalization bounds.
Multi-scale Online Learning and its Applications to Online Auctions
For the online posted pricing problem, we show regret bounds that scale with the best fixed price, rather than the range of the values.
Optimal algorithms for smooth and strongly convex distributed optimization in networks
For centralized (i. e. master/slave) algorithms, we show that distributing Nesterov's accelerated gradient descent is optimal and achieves a precision $\varepsilon > 0$ in time $O(\sqrt{\kappa_g}(1+\Delta\tau)\ln(1/\varepsilon))$, where $\kappa_g$ is the condition number of the (global) function to optimize, $\Delta$ is the diameter of the network, and $\tau$ (resp.
Kernel-based methods for bandit convex optimization
We consider the adversarial convex bandit problem and we build the first $\mathrm{poly}(T)$-time algorithm with $\mathrm{poly}(n) \sqrt{T}$-regret for this problem.
Black-box optimization with a politician
We propose a new framework for black-box convex optimization which is well-suited for situations where gradient computations are expensive.
Multi-scale exploration of convex functions and bandit convex optimization
We construct a new map from a convex function to a distribution on its domain, with the property that this distribution is a multi-scale exploration of the function.
Sampling from a log-concave distribution with Projected Langevin Monte Carlo
We extend the Langevin Monte Carlo (LMC) algorithm to compactly supported measures via a projection step, akin to projected Stochastic Gradient Descent (SGD).
A geometric alternative to Nesterov's accelerated gradient descent
The new algorithm has a simple geometric interpretation, loosely inspired by the ellipsoid method.
Bandit Convex Optimization: sqrt{T} Regret in One Dimension
We analyze the minimax regret of the adversarial bandit convex optimization problem.
The entropic barrier: a simple and optimal universal self-concordant barrier
We prove that the Cram\'er transform of the uniform measure on a convex body in $\mathbb{R}^n$ is a $(1+o(1)) n$-self-concordant barrier, improving a seminal result of Nesterov and Nemirovski.
Convex Optimization: Algorithms and Complexity
In stochastic optimization we discuss stochastic gradient descent, mini-batches, random coordinate descent, and sublinear algorithms.
Most Correlated Arms Identification
We study the problem of finding the most mutually correlated arms among many arms.
lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits
The paper proposes a novel upper confidence bound (UCB) procedure for identifying the arm with the largest mean in a multi-armed bandit game in the fixed confidence setting using a small number of total samples.
Prior-free and prior-dependent regret bounds for Thompson Sampling
Building on the techniques of Audibert and Bubeck [2009] and Russo and Roy [2013] we first show that Thompson Sampling attains an optimal prior-free bound in the sense that for any prior distribution its Bayesian regret is bounded from above by $14 \sqrt{n K}$.
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems
Multi-armed bandit problems are the most basic examples of sequential decision problems with an exploration-exploitation trade-off.
Regret in Online Combinatorial Optimization
We also recover the optimal bounds for the full information setting.
Multi-Bandit Best Arm Identification
We first propose an algorithm called Gap-based Exploration (GapE) that focuses on the arms whose mean is close to the mean of the best arm in the same bandit (i. e., small gap).
Online Optimization in X-Armed Bandits
We consider a generalization of stochastic bandit problems where the set of arms, X, is allowed to be a generic topological space.
