Search Results for author:
Found 4 papers, 0 papers with code
Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
We achieve this for the LLaMA-2 7B model by combining the SparseGPT one-shot pruning method and sparse pretraining of those models on a subset of the SlimPajama dataset mixed with a Python subset of The Stack dataset.
oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes
In this paper, we introduce the range of oBERTa language models, an easy-to-use set of language models which allows Natural Language Processing (NLP) practitioners to obtain between 3. 8 and 24. 3 times faster models without expertise in model compression.
Sparse*BERT: Sparse Models Generalize To New tasks and Domains
Our experimentation shows that models that are pruned during pretraining using general domain masked language models can transfer to novel domains and tasks without extensive hyperparameter exploration or specialized approaches.
Contour location via entropy reduction leveraging multiple information sources
We introduce an algorithm to locate contours of functions that are expensive to evaluate.
