Search Results for author:
Found 27 papers, 14 papers with code
Lion Cub: Minimizing Communication Overhead in Distributed Lion
Communication overhead is a key challenge in distributed deep learning, especially on slower Ethernet interconnects, and given current hardware trends, communication is likely to become a major bottleneck.
VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
We present the V:N:M format, which enables the execution of arbitrary N:M ratios on SPTCs.
Cached Operator Reordering: A Unified View for Fast GNN Training
Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering.
STen: Productive and Efficient Sparsity in PyTorch
As deep learning models grow, sparsity is becoming an increasingly critical component of deep neural networks, enabling improved performance and reduced storage.
Performance Embeddings: A Similarity-based Approach to Automatic Performance Optimization
Performance optimization is an increasingly challenging but often repetitive task.
A Theory of I/O-Efficient Sparse Neural Network Inference
We can only process data that is stored in fast memory, which incurs data movement (input/output-operations, or I/Os) between the two units.
ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts
We propose the ENS-10 prediction correction task for improving the forecast quality at a 48-hour lead time through ensemble post-processing.
A Data-Centric Optimization Framework for Machine Learning
Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute.
Learning Combinatorial Node Labeling Algorithms
We present a novel neural architecture to solve graph optimization problems where the solution consists of arbitrary node labels, allowing us to solve hard problems like graph coloring.
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components.
Clairvoyant Prefetching for Distributed Machine Learning I/O
I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments.
MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency
Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Deep Data Flow Analysis
Compiler architects increasingly look to machine learning when building heuristics for compiler optimization.
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Transformers are one of the most important machine learning workloads today.
Deep Learning for Post-Processing Ensemble Weather Forecasts
Applied to global data, our mixed models achieve a relative improvement in ensemble forecast skill (CRPS) of over 14%.
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging
For evaluation, we train ResNet-50 on ImageNet; Transformer for machine translation; and deep reinforcement learning for navigation at scale.
ProGraML: Graph-based Deep Learning for Program Optimization and Analysis
We introduce ProGraML - Program Graphs for Machine Learning - a novel graph-based program representation using a low level, language agnostic, and portable format; and machine learning models capable of performing complex downstream tasks over these graphs.
Predicting Weather Uncertainty with Deep Convnets
Modern weather forecast models perform uncertainty quantification using ensemble prediction systems, which collect nonparametric statistics based on multiple perturbed simulations.
Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency
Although trained on images of aspecific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself.
Stateful Dataflow Multigraphs: A Data-Centric Model for High-Performance Parallel Programs
With the ubiquity of accelerators, such as FPGAs and GPUs, the complexity of high-performance programming is increasing beyond the skill-set of the average scientist in domains outside of computer science.
Programming Languages Distributed, Parallel, and Cluster Computing Performance
Graph Processing on FPGAs: Taxonomy, Survey, Challenges
To facilitate understanding of this emerging domain, we present the first survey and taxonomy on graph computations on FPGAs.
Distributed, Parallel, and Cluster Computing Hardware Architecture
A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning
We introduce Deep500: the first customizable benchmarking infrastructure that enables fair comparison of the plethora of deep learning frameworks, algorithms, libraries, and techniques.
Augment your batch: better training with larger batches
We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets.
Neural Code Comprehension: A Learnable Representation of Code Semantics
In this paper, we propose a novel processing technique to learn code semantics, and apply it to a variety of program analysis tasks.
μ-cuDNN: Accelerating Deep Learning Frameworks with Micro-Batching
NVIDIA cuDNN is a low-level library that provides GPU kernels frequently used in deep learning.
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning.
