Search Results for author:
Found 22 papers, 18 papers with code
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144.
Petals: Collaborative Inference and Fine-tuning of Large Models
However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
Many recent breakthroughs in deep learning were achieved by training increasingly larger models on massive datasets.
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes.
Distributed Deep Learning in Open Collaborations
Modern deep learning applications require increasingly more compute to train state-of-the-art models.
Secure Distributed Training at Scale
Training such models requires a lot of computational resources (e. g., HPC clusters) that are not available to small research groups and independent researchers.
Training Transformers Together
The infrastructure necessary for training state-of-the-art models is becoming overly expensive, which makes training such models affordable only to large corporations and institutions.
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding.
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
Many deep learning applications benefit from using large models with billions of parameters.
Embedding Words in Non-Vector Space with Unsupervised Graph Learning
We adopt a recent method learning a representation of data in the form of a differentiable weighted graph and use it to modify the GloVe training algorithm.
RuCoLA: Russian Corpus of Linguistic Acceptability
Linguistic acceptability (LA) attracts the attention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering implausible texts with acceptability classifiers.
Ranked #2 on
Linguistic Acceptability
on ItaCoLA
Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics
Text-to-image generation models represent the next step of evolution in image synthesis, offering a natural way to achieve flexible yet fine-grained control over the result.
It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning
Commonsense reasoning is one of the key problems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English.
Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets
\emph{Ensemble Distribution Distillation} is an approach that allows a single model to efficiently capture both the predictive performance and uncertainty estimates of an ensemble.
Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy
Text-to-image synthesis has recently attracted widespread attention due to rapidly improving quality and numerous practical applications.
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples.
Unsupervised Discovery of Interpretable Latent Manipulations in Language VAEs
In this work, we make the first step towards unsupervised discovery of interpretable directions in language latent spaces.
Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality.
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters.
The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
Large Language Models (LLMs) have transformed the Natural Language Processing (NLP) landscape with their remarkable ability to understand and generate human-like text.
