Search Results for author:
Found 19 papers, 10 papers with code
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs.
DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference
Decoding using tree search can greatly enhance the inference quality for transformer-based Large Language Models (LLMs).
Exploring the Robustness of Decentralized Training for Large Language Models
Decentralized training of large language models has emerged as an effective way to democratize this technology.
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
We show that contextual sparsity exists, that it can be accurately predicted, and that we can exploit it to speed up LLM inference in wall-clock time without compromising LLM's quality or in-context learning ability.
Serving Deep Learning Model in Relational Databases
Serving deep learning (DL) models on relational data has become a critical requirement across diverse commercial and scientific domains, sparking growing interest recently.
Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning
The relational data model was designed to facilitate large-scale data management and analytics.
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
Stochastic Gradient Descent without Full Data Shuffle
In this paper, we first conduct a systematic empirical study on existing data shuffling strategies, which reveals that all existing strategies have room for improvement -- they all suffer in terms of I/O performance or convergence rate.
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks.
Decentralized Training of Foundation Models in Heterogeneous Environments
Our key technical contribution is a scheduling algorithm that allocates different computational "tasklets" in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network.
Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters
Specifically, in order to ensure both the training efficiency and the training accuracy, we design a novel hybrid training algorithm, where the embedding layer and the dense neural network are handled by different synchronization mechanisms; then we build a system called Persia (short for parallel recommendation training system with hybrid acceleration) to support this hybrid training algorithm.
BAGUA: Scaling up Distributed Learning with System Relaxations
Recent years have witnessed a growing list of systems for distributed data-parallel training.
Tensor Relational Algebra for Machine Learning System Design
This implementation abstraction provides little built-in support for ML systems to scale past a single machine, or for handling large models with matrices or tensors that do not easily fit into the RAM of an ASIC.
A Federated Learning Framework for Healthcare IoT devices
The Internet of Things (IoT) revolution has shown potential to give rise to many medical applications with access to large volumes of healthcare data collected by IoT devices.
Distributed Learning of Deep Neural Networks using Independent Subnet Training
These properties of IST can cope with issues due to distributed data, slow interconnects, or limited device memory, making IST a suitable approach for cases of mandatory distribution.
Diagnosing Cardiac Abnormalities from 12-Lead Electrocardiograms Using Enhanced Deep Convolutional Neural Networks
We train an enhanced deep convolutional neural network in order to identify eight cardiac abnormalities from the standard 12-lead electrocardiograms (ECGs) using the dataset of 14000 ECGs.
Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
A number of popular systems, most notably Google's TensorFlow, have been implemented from the ground up to support machine learning tasks.
WaveletAE: A Wavelet-enhanced Autoencoder for Wind Turbine Blade Icing Detection
Quick detection of blade ice accretion is crucial for the maintenance of wind farms.
