Search Results for author:
Found 6 papers, 3 papers with code
Scaling Laws for Fine-Grained Mixture of Experts
Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget.
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers.
Structured Packing in LLM Training Improves Long Context Utilization
Recent developments in long-context large language models have attracted considerable attention.
Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation
The operation of matching experts and tokens is discrete, which makes MoE models prone to issues like training instability and uneven expert utilization.
Sparse is Enough in Scaling Transformers
We study sparse variants for all layers in the Transformer and propose Scaling Transformers, a family of next generation Transformer models that use sparse layers to scale efficiently and perform unbatched decoding much faster than the standard Transformer as we scale up the model size.
Neural heuristics for SAT solving
We use neural graph networks with a message-passing architecture and an attention mechanism to enhance the branching heuristic in two SAT-solving algorithms.
