Search Results for author:
Found 6 papers, 3 papers with code
ARB: Advanced Reasoning Benchmark for Large Language Models
As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint.
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Multi-modal language-vision models trained on hundreds of millions of image-text pairs (e. g.
Current Limitations of Language Models: What You Need is Retrieval
We classify and re-examine some of the current approaches to improve the performance-computes trade-off of language models, including (1) non-causal models (such as masked language models), (2) extension of batch length with efficient attention, (3) recurrence, (4) conditional computation and (5) retrieval.
One Epoch Is All You Need
We compare the wall-clock time of the training of models with different parameter budget under one epoch training, and we show that size/iteration adjustment based on our proposed heuristics leads to 1-2. 7x speedup in our cases.
Extractive Summary as Discrete Latent Variables
In this paper, we compare various methods to compress a text using a neural model.
