Search Results for author:
Found 5 papers, 4 papers with code
Scaling Laws for Fine-Grained Mixture of Experts
Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget.
Mixtral of Experts
In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
Ranked #9 on
Question Answering
on PIQA
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers.
Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation
The operation of matching experts and tokens is discrete, which makes MoE models prone to issues like training instability and uneven expert utilization.
Magnushammer: A Transformer-Based Approach to Premise Selection
By combining \method with a language-model-based automated theorem prover, we further improve the state-of-the-art proof success rate from $57. 0\%$ to $71. 0\%$ on the PISA benchmark using $4$x fewer parameters.
