Search Results for author:
Found 22 papers, 12 papers with code
Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers
In this work, we study the effectiveness of existing sparse training recipes at \textit{high-sparsity regions} and argue that these methods fail to sustain the model quality on par with low-sparsity regions.
Scaling Laws for Sparsely-Connected Foundation Models
We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i. e., "foundation models"), in both vision and language domains.
Dynamic Sparse Training with Structured Sparsity
Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference.
JaxPruner: A concise library for sparsity research
This paper introduces JaxPruner, an open-source JAX-based pruning and sparse training library for machine learning research.
The Dormant Neuron Phenomenon in Deep Reinforcement Learning
In this work we identify the dormant neuron phenomenon in deep reinforcement learning, where an agent's network suffers from an increasing number of inactive neurons, thereby affecting network expressivity.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ObjectNet
Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask
In this work, we focus on N:M sparsity and extensively study and evaluate various training recipes for N:M sparsity in terms of the trade-off between model accuracy and compute cost (FLOPs).
The State of Sparse Training in Deep Reinforcement Learning
The use of sparse neural networks has seen rapid growth in recent years, particularly in computer vision.
GradMax: Growing Neural Networks using Gradient Information
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified.
Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning
We propose a method, Head-to-Toe probing (Head2Toe), that selects features from all layers of the source model to train a classification head for the target-domain.
Head2Toe: Utilizing Intermediate Representations for Better OOD Generalization
We propose a method, Head-to-Toe probing (Head2Toe), that selects features from all layers of the source model to train a classification head for the target-domain.
Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
To bridge this gap, we perform a cross-family study of the best transfer and meta learners on both a large-scale meta-learning benchmark (Meta-Dataset, MD), and a transfer learning benchmark (Visual Task Adaptation Benchmark, VTAB).
Practical Real Time Recurrent Learning with a Sparse Approximation
For highly sparse networks, SnAp with $n=2$ remains tractable and can outperform backpropagation through time in terms of learning speed when updates are done online.
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win
Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training.
A Practical Sparse Approximation for Real Time Recurrent Learning
Current methods for training recurrent neural networks are based on backpropagation through time, which requires storing a complete history of network states, and prohibits updating the weights `online' (after every timestep).
Rigging the Lottery: Making All Tickets Winners
There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model.
Ranked #1 on
Sparse Learning
on ImageNet
Natural Language Understanding with the Quora Question Pairs Dataset
This paper explores the task Natural Language Understanding (NLU) by looking at duplicate question detection in the Quora dataset.
The Difficulty of Training Sparse Neural Networks
Additionally, our attempts to find a decreasing objective path from "bad" solutions to the "good" ones in the sparse subspace fail.
Mean Replacement Pruning
Finally, we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training.
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples
Few-shot classification refers to learning a classifier for new classes given only a few examples.
Ranked #7 on
Few-Shot Image Classification
on Meta-Dataset Rank
Detecting Dead Weights and Units in Neural Networks
We propose an efficient way for detecting dead units and use it to select which units to prune.
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.
