Search Results for author:
Found 45 papers, 21 papers with code
MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache
How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements.
Transforming the Hybrid Cloud for Emerging AI Workloads
This white paper, developed through close collaboration between IBM Research and UIUC researchers within the IIDAI Institute, envisions transforming hybrid cloud systems to meet the growing complexity of AI workloads through innovative, full-stack co-design approaches, emphasizing usability, manageability, affordability, adaptability, efficiency, and scalability.
Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment
Safety alignment of Large Language Models (LLMs) has recently become a critical objective of model developers.
Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks
To mitigate computational costs, LLMs often employ the KV Cache technique to improve the generation speed.
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
This paper presents UltraEdit, a large-scale (approximately 4 million editing samples), automatically generated dataset for instruction-based image editing.
Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training
First, the universal checkpoint format, which consists of a consolidated representation of each model parameter and metadata for mapping parameter fragments into training ranks of arbitrary model-parallelism configuration.
Computing in the Era of Large Generative Models: From Cloud-Native to AI-Native
In this paper, we investigate the intersection of large generative AI models and cloud-native computing architectures.
Configuration Validation with Large Language Models
We present a first analysis on the feasibility and effectiveness of using LLMs for configuration validation.
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences.
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs).
DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention
Most of the existing multi-modal models, hindered by their incapacity to adeptly manage interleaved image-and-text inputs in multi-image, multi-round dialogues, face substantial constraints in resource allocation for training and data accessibility, impacting their adaptability and scalability across varied interaction realms.
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
Computation in a typical Transformer-based large language model (LLM) can be characterized by batch size, hidden dimension, number of layers, and sequence length.
RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model
Text-to-image generation (TTI) refers to the usage of models that could process text input and generate high fidelity images based on text descriptions.
Cost-effective On-device Continual Learning over Memory Hierarchy with Miro
Continual learning (CL) trains NN models incrementally from a continuous stream of tasks.
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
ChatGPT-like models have revolutionized various applications in artificial intelligence, from summarization and coding to translation, matching or even surpassing human performance.
DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
Compared to the rapidly evolving model architecture, how to efficiently use the training data (especially for the expensive foundation model pretraining) is both less explored and difficult to realize due to the lack of a convenient framework that focuses on data efficiency capabilities.
Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers
Large-scale transformer models have become the de-facto architectures for various machine learning applications, e. g., CV and NLP.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
DeepSpeed Inference reduces latency by up to 7. 3X over the state-of-the-art for latency-oriented scenarios and increases throughput by over 1. 5x for throughput-oriented scenarios.
Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding
Furthermore, we present and inexpensive, heuristic-driven search algorithm that identifies promising heterogeneous compression configurations that meet a compression ratio constraint.
Extreme Compression for Pre-trained Transformers Made Simple and Efficient
Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices.
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements.
Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
DNN models across many domains continue to grow in size, resulting in high resource requirements for effective training, and unpalatable (and often unaffordable) costs for organizations and research labs across scales.
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
1-bit gradient compression and local steps are two representative techniques that enable drastic communication reduction in distributed SGD.
ScaLA: Accelerating Adaptation of Pre-Trained Transformer-Based Language Models via Efficient Large-Batch Adversarial Noise
In recent years, large pre-trained Transformer-based language models have led to dramatic improvements in many natural language understanding tasks.
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model.
A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities
With both scaling trends, new problems and challenges emerge in DL inference serving systems, which gradually trends towards Large-scale Deep learning Serving systems (LDS).
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
In particular, we propose to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration.
Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
In particular, in mobile and IoT devices, real-time data can be stored not just in high-speed RAMs but in internal storage devices as well, which offer significantly larger capacity than the RAMs.
ScaLA: Speeding-Up Fine-tuning of Pre-trained Transformer Networks via Efficient and Scalable Adversarial Perturbation
To address this challenge, we propose ScaLA, a scalable and robust method for large-batch optimization of transformer networks via adversarial perturbation.
HoloFormer: Deep Compression of Pre-Trained Transforms via Unified Optimization of N:M Sparsity and Integer Quantization
In this work, we propose a unified, systematic approach to learning N:M sparsity and integer quantization for pre-trained Transformers using the Alternating Directions Method of Multipliers (ADMM).
Demystifying Hyperparameter Optimization in Federated Learning
Such data heterogeneity and privacy requirements bring unique challenges for learning hyperparameter optimization as the training dynamics change across clients even within the same training round and they are difficult to measure due to privacy constraints.
The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
To reduce the wall-clock training time, a common practice is to increase the batch size and learning rate.
Understanding and Generalizing Monotonic Proximity Graphs for Approximate Nearest Neighbor Search
Our experiments give guidance on how to approximate and generalize MRNG to build proximity graphs on a large scale.
ZeRO-Offload: Democratizing Billion-Scale Model Training
By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.
DynaTune: Dynamic Tensor Program Optimization in Deep Neural Network Compilation
Recently, the DL compiler, together with Learning to Compile has proven to be a powerful technique for optimizing deep learning models.
AdaTune: Adaptive Tensor Program Compilation Made Efficient
Deep learning models are computationally intense, and implementations often have to be highly optimized by experts or hardware vendors to be usable in practice.
HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory
The emergence of heterogeneous memory (HM) brings a solution to significantly increase memory capacity and break the above tradeoff: Using HM, billions of data points can be placed in the main memory on a single machine without using any data compression.
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains.
SHARP: An Adaptable, Energy-Efficient Accelerator for Recurrent Neural Network
To solve these issues, we propose an intelligent tiled-based dispatching mechanism for increasing the adaptiveness of RNN computation, in order to efficiently handle the data dependencies.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Learning to Anneal and Prune Proximity Graphs for Similarity Search
This paper studies similarity search, which is a crucial enabler of many feature vector--based applications.
Zoom: SSD-based Vector Search for Optimizing Accuracy, Latency and Memory
With the advancement of machine learning and deep learning, vector search becomes instrumental to many information retrieval systems, to search and find best matches to user queries based on their semantic similarities. These online services require the search architecture to be both effective with high accuracy and efficient with low latency and memory footprint, which existing work fails to offer.
Navigating with Graph Representations for Fast and Scalable Decoding of Neural Language Models
Neural language models (NLMs) have recently gained a renewed interest by achieving state-of-the-art performance across many natural language processing (NLP) tasks.
Learning Intrinsic Sparse Structures within Long Short-Term Memory
This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs.
