Search Results for author:
Found 13 papers, 5 papers with code
TinyGSM: achieving >80% on GSM8k with small language models
Specifically for solving grade school math, the smallest model size so far required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Ranked #58 on
Arithmetic Reasoning
on GSM8K
How to Fine-Tune Vision Models with SGD
SGD and AdamW are the two most used optimizers for fine-tuning large neural networks in computer vision.
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Furthermore, existing MoE works do not consider computational constraints (e. g., FLOPs, latency) to guide their design.
LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
Results show that the perplexity of 16-layer GPT-2 and Transformer-XL can be achieved with up to 1. 5x, 2. 5x faster runtime and 1. 2x, 2. 0x lower peak memory utilization.
Ranking Convolutional Architectures by their Feature Extraction Capabilities
We propose a simple but powerful method which we call FEAR, for ranking architectures in any search space.
FEAR: A Simple Lightweight Method to Rank Architectures
We propose a simple but powerful method which we call FEAR, for ranking architectures in any search space.
Ranking Architectures by Feature Extraction Capabilities
By training different architectures in the search space to the same training or validation error and subsequently comparing the usefulness of the features extracted on the task-dataset of interest by freezing most of the architecture we obtain quick estimates of the relative performance.
Network size and size of the weights in memorization with two-layers neural networks
In contrast we propose a new training procedure for ReLU networks, based on {\em complex} (as opposed to {\em real}) recombination of the neurons, for which we show approximate memorization with both $O\left(\frac{n}{d} \cdot \frac{\log(1/\epsilon)}{\epsilon}\right)$ neurons, as well as nearly-optimal size of the weights.
Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
In this paper, we employ adversarial training to improve the performance of randomized smoothing.
Is Q-learning Provably Efficient?
We prove that, in an episodic MDP setting, Q-learning with UCB exploration achieves regret $\tilde{O}(\sqrt{H^3 SAT})$, where $S$ and $A$ are the numbers of states and actions, $H$ is the number of steps per episode, and $T$ is the total number of steps.
Finite-Time Analysis of Projected Langevin Monte Carlo
We analyze the projected Langevin Monte Carlo (LMC) algorithm, a close cousin of projected Stochastic Gradient Descent (SGD).
On Finding the Largest Mean Among Many
Motivated by large-scale applications, we are especially interested in identifying situations where the total number of samples that are necessary and sufficient to find the best arm scale linearly with the number of arms.
Optimal discovery with probabilistic expert advice: finite time analysis and macroscopic optimality
We consider an original problem that arises from the issue of security analysis of a power system and that we name optimal discovery with probabilistic expert advice.
