Search Results for author:
Found 27 papers, 16 papers with code
TBD: Benchmarking and Analyzing Deep Neural Network Training
Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine).
Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
For each feature map recomputation to be effective and efficient, its effect on (1) the total memory footprint, and (2) the total execution time has to be carefully estimated.
StreamBox-HBM: Stream Analytics on High Bandwidth Hybrid Memory
It dynamically optimizes for both the high bandwidth and limited capacity of HBM, and the limited bandwidth and high capacity of standard DRAM.
Databases
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Priority-based Parameter Propagation for Distributed DNN Training
Data parallel training is widely used for scaling distributed deep neural network (DNN) training.
BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
In an era when the performance of a single compute device plateaus, software must be designed to scale on massively parallel systems for better runtime performance.
MLPerf Training Benchmark
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML.
Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
Modern deep neural network (DNN) training jobs use complex and heterogeneous software/hardware stacks.
Multi-node Bert-pretraining: Cost-efficient Approach
As a result, to train these models within a reasonable time, machine learning (ML) programmers often require advanced hardware setups such as the premium GPU-enabled NVIDIA DGX workstations or specialized accelerators such as Google's TPU Pods.
Skyline: Interactive In-Editor Computational Performance Profiling for Deep Neural Network Training
Training a state-of-the-art deep neural network (DNN) is a computationally-expensive and time-consuming process, which incentivizes deep learning developers to debug their DNNs for computational performance.
TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
TensorDash is a hardware level technique for enabling data-parallel MAC units to take advantage of sparsity in their input operand streams.
FPRaker: A Processing Element For Accelerating Neural Network Training
We demonstrate that FPRaker can be used to compose an accelerator for training and that it can improve performance and energy efficiency compared to using conventional floating-point units under ISO-compute area constraints.
IOS: Inter-Operator Scheduler for CNN Acceleration
To accelerate CNN inference, existing deep learning frameworks focus on optimizing intra-operator parallelization.
A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
Our technique exploits the observation that, because DNN training consists of repetitive compute steps, predicting the execution time of a single iteration is usually enough to characterize the performance of an entire training process.
Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
Driven by the tremendous effort in researching novel deep learning (DL) algorithms, the training cost of developing new models increases staggeringly in recent years.
RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications.
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes.
Distributed Deep Learning in Open Collaborations
Modern deep learning applications require increasingly more compute to train state-of-the-art models.
MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience.
Optimizing Data Collection in Deep Reinforcement Learning
We show that simulator kernel fusion speedups with a simple simulator are $11. 3\times$ and increase by up to $1024\times$ as simulator complexity increases in terms of memory bandwidth requirements.
Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs
With the proposed paradigm, we implement a deep learning compiler Hidet.
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
We implement Tempo and evaluate the throughput, memory usage, and accuracy/loss on the BERT Large pre-training task.
Speeding up Fourier Neural Operators via Mixed Precision
In this work, we (i) profile memory and runtime for FNO with full and mixed-precision training, (ii) conduct a study on the numerical stability of mixed-precision training of FNO, and (iii) devise a training routine which substantially decreases training time and memory usage (up to 34%), with little or no reduction in accuracy, on the Navier-Stokes and Darcy flow equations.
The Synergy of Speculative Decoding and Batching in Serving Large Language Models
Large Language Models (LLMs) like GPT are state-of-the-art text generation models that provide significant assistance in daily routines.
Minuet: Accelerating 3D Sparse Convolutions on GPUs
Minuet proposes to (i) replace the hash tables used in the Map step with a novel segmented sorting double-traversed binary search algorithm that highly utilizes the on-chip memory hierarchy of GPUs, (ii) use a lightweight scheme to autotune the tile size in the Gather and Scatter operations of the GMaS step, such that to adapt the execution to the particular characteristics of each SC layer, dataset, and GPU architecture, and (iii) employ a padding-efficient GEMM grouping approach that reduces both memory padding and kernel launching overheads.
Accelerating Graph Neural Networks on Real Processing-In-Memory Systems
Graph Neural Network (GNN) execution involves both compute-intensive and memory-intensive kernels, the latter dominates the total time, being significantly bottlenecked by data movement between memory and processors.
