Search Results for author:
Found 5 papers, 1 papers with code
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
In this work, we propose Hydra heads, a sequentially dependent, drop-in replacement for standard draft heads that significantly improves speculation accuracy.
Striped Attention: Faster Ring Attention for Causal Transformers
In experiments running Striped Attention on A100 GPUs and TPUv4s, we are able to achieve up to 1. 45x end-to-end throughput improvements over the original Ring Attention algorithm on causal transformer training at a sequence length of 256k.
Dynamic Masking Rate Schedules for MLM Pretraining
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%.
The Effect of Data Dimensionality on Neural Network Prunability
Practitioners prune neural networks for efficiency gains and generalization improvements, but few scrutinize the factors determining the prunability of a neural network the maximum fraction of weights that pruning can remove without compromising the model's test accuracy.
3D Neural Field Generation using Triplane Diffusion
Diffusion models have emerged as the state-of-the-art for image generation, among other tasks.
