Search Results for author:
Found 14 papers, 9 papers with code
Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning
The relational data model was designed to facilitate large-scale data management and analytics.
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
Stochastic Gradient Descent without Full Data Shuffle
In this paper, we first conduct a systematic empirical study on existing data shuffling strategies, which reveals that all existing strategies have room for improvement -- they all suffer in terms of I/O performance or convergence rate.
Decentralized Training of Foundation Models in Heterogeneous Environments
Our key technical contribution is a scheduling algorithm that allocates different computational "tasklets" in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network.
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks.
Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters
Specifically, in order to ensure both the training efficiency and the training accuracy, we design a novel hybrid training algorithm, where the embedding layer and the dense neural network are handled by different synchronization mechanisms; then we build a system called Persia (short for parallel recommendation training system with hybrid acceleration) to support this hybrid training algorithm.
BAGUA: Scaling up Distributed Learning with System Relaxations
Recent years have witnessed a growing list of systems for distributed data-parallel training.
Tensor Relational Algebra for Machine Learning System Design
This implementation abstraction provides little built-in support for ML systems to scale past a single machine, or for handling large models with matrices or tensors that do not easily fit into the RAM of an ASIC.
A Federated Learning Framework for Healthcare IoT devices
The Internet of Things (IoT) revolution has shown potential to give rise to many medical applications with access to large volumes of healthcare data collected by IoT devices.
Distributed Learning of Deep Neural Networks using Independent Subnet Training
These properties of IST can cope with issues due to distributed data, slow interconnects, or limited device memory, making IST a suitable approach for cases of mandatory distribution.
Diagnosing Cardiac Abnormalities from 12-Lead Electrocardiograms Using Enhanced Deep Convolutional Neural Networks
We train an enhanced deep convolutional neural network in order to identify eight cardiac abnormalities from the standard 12-lead electrocardiograms (ECGs) using the dataset of 14000 ECGs.
Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
A number of popular systems, most notably Google's TensorFlow, have been implemented from the ground up to support machine learning tasks.
WaveletAE: A Wavelet-enhanced Autoencoder for Wind Turbine Blade Icing Detection
Quick detection of blade ice accretion is crucial for the maintenance of wind farms.
