Search Results for author:
Found 20 papers, 5 papers with code
Microscaling Data Formats for Deep Learning
Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications.
AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks
We integrated GA with the latest learnable pruning methods to create an automated sparse training algorithm called AutoSparse, which achieves better accuracy and/or training/inference FLOPS reduction than existing learnable pruning methods for sparse ResNet50 and MobileNetV1 on ImageNet-1K: AutoSparse achieves (2x, 7x) reduction in (training, inference) FLOPS for ResNet50 on ImageNet at 80% sparsity.
FP8 Formats for Deep Learning
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors.
Systolic Computing on GPUs for Productive Performance
We propose a language and compiler to productively build high-performance {\it software systolic arrays} that run on GPUs.
(Updated submission 11/20/2020) MISIM: A Novel Code Similarity System
First, MISIM uses a novel context-aware semantic structure (CASS), which is designed to aid in lifting semantic meaning from code syntax.
MISIM: A Neural Code Semantics Similarity System Using the Context-Aware Semantics Structure
Code semantics similarity can be used for many tasks such as code recommendation, automated software defect correction, and clone detection.
Context-Aware Parse Trees
The simplified parse tree (SPT) presented in Aroma, a state-of-the-art code recommendation system, is a tree-structured representation used to infer code semantics by capturing program \emph{structure} rather than program \emph{syntax}.
K-TanH: Efficient TanH For Deep Learning
We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning.
A Study of BFLOAT16 for Deep Learning Training
In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017).
On Scale-out Deep Learning Training for Cloud and HPC
The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes.
Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
This paper presents the first, 15-PetaFLOP Deep Learning system for solving scientific pattern classification problems on contemporary HPC architectures.
Ternary Residual Networks
Aided by such an elegant trade-off between accuracy and compute, the 8-2 model (8-bit activations, ternary weights), enhanced by ternary residual edges, turns out to be sophisticated enough to achieve very high accuracy ($\sim 1\%$ drop from our FP-32 baseline), despite $\sim 1. 6\times$ reduction in model size, $\sim 26\times$ reduction in number of multiplications, and potentially $\sim 2\times$ power-performance gain comparing to 8-8 representation, on the state-of-the-art deep network ResNet-101 pre-trained on ImageNet dataset.
Ternary Neural Networks with Fine-Grained Quantization
We address this by fine-tuning Resnet-50 with 8-bit activations and ternary weights at $N=64$, improving the Top-1 accuracy to within $4\%$ of the full precision result with $<30\%$ additional training overhead.
Parallelizing Word2Vec in Multi-Core and Many-Core Architectures
Word2vec is a widely used algorithm for extracting low-dimensional vector representations of words.
Faster CNNs with Direct Sparse Convolutions and Guided Pruning
Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels.
Parallelizing Word2Vec in Shared and Distributed Memory
In combination, these techniques allow us to scale up the computation near linearly across cores and nodes, and process hundreds of millions of words per second, which is the fastest word2vec implementation to the best of our knowledge.
Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior.
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
One way to understand BlackOut is to view it as an extension of the DropOut strategy to the output layer, wherein we use a discriminative training loss and a weighted sampling scheme.
